Paper: Differentially Private Low-dimensional Synthetic Data from High-dimensional Datasets (ICLR-2024)

Reviewer comment (Theory):
The "private covariance matrix" and "private linear projection" steps in Algorithm 1 should have $\epsilon / 4$, not $\epsilon / 2$.

Author response:
Thank you for pointing out the mistake. Indeed, by Laplacian mechanism, to protect $\overline{X}\in \mathbb{R}^d$ with parameter $\varepsilon$ needs entry-wise noises of $\mathrm{Lap}(\frac{d}{\varepsilon n})$, and hence the private mean estimation would have a slightly worse accuracy bound $O(d/n)$. Together with Weakness 2, we will change the privacy budget of each step in Algorithm 1 to have a correct level of privacy, and we will update the result in the simulation experiments. Nevertheless, it would only influence the final result by $O(d/n)$, and this is an inconsequential term compared with the dominating term $n^{-1/d'}$. Therefore, the final accuracy bound Theorem 4.2 for the main low-dimensional synthetic data algorithm remains to be true.

Reviewer comment (Theory):
You claim that the Wasserstein distance bound gives accuracy guarantees for many machine learning algorithms. How strong are these guarantees in practice? For example, what is the accuracy guarantee for the SVM in the experiment of the paper (if there is one)?

Author response:
(1) We compare the accuracy of SVM when applied on the true and synthetic data separately. Our experiment suggests the low-dimensional synthetic data approach is useful for classification tasks.

The theoretical utility of Wasserstein bound is guaranteed by the Kantorovich-Rubinstein duality, which ensures the accuracy of all the Lipschitz tasks. (See the last equation in Appendix A.)  As there are alternative variations of SVM, the practical guarantee is out of the scope for this paper.

Reviewer comment (Experiments):
What is the accuracy of the SVM on the real data, without DP?

Author response:
(2) Without DP requirement, the accuracy of SVM is around 95%. 
    
The experiment in the paper can be seen as a proof of concept for the Wasserstein accuracy bound we proved. As a result, the low-dimensional algorithm does overcome the curse of high-dimensionality.  Optimizing the algorithm more practically could potentially increase SVM's performance. However, it is not the main focus of our work in this paper.

Reviewer comment (Theory):
Assuming that the data lies close to a low-dimensional linear subspace (instead of a more general manifold) sounds very restrictive. Do you have some idea on what settings would this assumption be realistic in?

Author response:
Thank you for the response. The linear subspace assumptions are common in various foundational theoretical work, such as PCA and linear regression. The main focus of our paper is to prove the theoretical bound of synthetic data accuracy. As a result, with the idea of low-dimensional subspace projection, we deduce a near optimal bound where $n^{-1/d'}$ term is necessary.

Reviewer comment (Writing):
Is $\sigma_i(M)$ the $i$-th largest eigenvalue in Theorem 1.2, or is the order something else?

Author response:
(3) $\sigma_i(M)$ denotes the $i$-th largest singular value of $M$. Here $M$ is semi-positive-definite, so it is also the $i$-th largest eigenvalue.

Reviewer comment (Experiments):
How could $d'$ be chosen in practice?

Author response:
(4) Thank you for the question. We mentioned the *Adaptive and private choices of $d'$* at Section 1.1, but we omit the details of such optimal choice due to space limitation. We will add the details of the choice of $d'$ in Appendix E. The main idea for the choice of $d'$ is to balance different error terms  Theorem 4.2; we can choose the best $d'$ to minimize the error. Such a minimization problem would be time-efficient as we only need the values of the singular values of the private covariance matrix.

Reviewer comment (Experiments):
The experimental against the full-dimension algorithm is convincing, but I also would like to see how it perform against the method of Donhauser et al. (2023), especially when the authors claim that their method is more accurate than that of the previous work.

Author response:
(1)  The method of Donhauser et al. (20203) uses the exponential mechanism and it is not very efficient to implement. Our approach is computationally efficient and has a better theoretical guarantee in terms of the error rate. Our main focus is on the theoretical guarantees therefore we do not  implement their methods in this work.

Reviewer comment (Theory):
There should be a discussion on the $\sqrt{\sum_{i>d'}\sigma_i(M)}$ term, especially when $d'$ is unknown (which occurs in most use cases). How do we make sure that this term does not dominate the rest of the error bound?

Author response:
(2) We mentioned the *Adaptive and private choices of $d'$* at Section 1.1, but we omit the details of such optimal choice due to space limitation. The idea is exactly the gist of Question 2: to choose the best $d'$ privately by balancing different terms in the error bound in Theorem 4.2. (And as mentioned in the question, $\sigma_i(\widehat{\mathbf M})$'s are needed to estimate $d'$ privately.) Moreover, such optimization is time efficient. We will add the details of the choice of $d'$ in Appendix E.

Reviewer comment (Theory):
In the Conclusion, the authors claim that Donhauser et al. (2023)'s accuracy rate is $(\varepsilon n)^{-1/(d'+1)}$. I skimmed the said paper and couldn't found the exact bound. Can the authors point me to where the said bound is located?

Author response:
(1) The result of $(\varepsilon n)^{-1/(d'+1)}$ is from Theorem 3 in Donhauser et al. (2023). In their settings, the test function is also all the 1-Lipschitz functions, and the dataset is in a low-dimensional manifold with Minkowski dimension $d'$.

Reviewer comment (Theory):
I think the upper bound can be used to find an "optimal" value of $d'$ by comparing $\sqrt{\sum_{i>d'}\sigma_i(\hat{M})}$ and the rest of term (we also might have to take the bias of $\sigma_i(\hat{M})$ into account. Can the authors make some comments on this approach?

Author response:
(2) Thank you for the suggested approach. We mentioned the *Adaptive and private choices of $d'$* at Section 1.1, but we omit the details of such optimal choice due to space limitation. The idea is exactly the gist of Question 2: to choose the best $d'$ privately by balancing different terms in the error bound in Theorem 4.2. (And as mentioned in the question, $\sigma_i(\widehat{\mathbf M})$'s are needed to estimate $d'$ privately.) Moreover, such optimization is time efficient. We will add the details of the choice of $d'$ in Appendix E.

Reviewer comment (Writing):
In Algorithm 1: "Let $\overline{X}$ be the mean value of the dataset". Is this the original or the synthetic dataset?

Author response:
(1) This is the mean value of the original dataset.

Reviewer comment (Theory):
Continuing from above, if it is the mean of the original dataset, can't we just save the privatized mean from the Linear Projection step and add it back after Low-dimensional Synthetic Data? This would help save the privacy budget by $\varepsilon/4$.

Author response:
(2) Thank you for the suggestion. Saving the private mean and using it afterward would have better accuracy than adding noises twice. We changed the algorithm to have a more reasonable privacy budget distribution. Nevertheless, it will only influence the final accuracy bound by $O(1/n)$, which is dominated by the $n^{-1/d'}$ term.

Reviewer comment (Theory):
In Algorithm 1, I see two mechanisms with privacy budget $\varepsilon/2$, and two with $\varepsilon/4$. I am not sure if I am interpreting this correctly since total privacy budget is larger than $\varepsilon$.

Author response:
(3) Thank you for pointing it out. Combining with the suggestion in Minor Question 2, we reassign the privacy budget in Algorithm 1 in the correct way.

Reviewer comment (Reproducibility):
In Algorithm 2, the definition of $\boldsymbol{A}_{ij}$ when $i>j$ is missing.

Author response:
(4) In the algorithm, we define the matrix $A$ to be symmetric, so the entries below the diagonal are implied from the other entries. We will make it clearer in the modification.

Reviewer comment (Theory):
The expected Wasserstein distance is $poly(d)$. The second term is still alright if we have enough samples, but the third term is the one that concerns me. When $d$ is large, the third term does not really help a lot, especially when $d'$ is not too small. That said, the second term becomes too large if $d'$ is large, so there seems to be quite a trade-off over there.

Author response:
Yes, there would be a trade-off, and it is possible to choose a better value of $d'$ privately. We mentioned the *Adaptive and private choices of $d'$* at Section 1.1, but we omit the details of such optimal choice due to space limitation. We will add the details of the choice of $d'$ in the appendix. The main idea for the choice of $d'$ is to balance different terms in the error bound of Theorem 4.2, and we can choose the best $d'$ to minimize the error. Such a minimization problem would be time-efficient as we only need the values of the singular values of the private covariance matrix.

For a non-trivial bound of Wasserstein distance, we need the number of data $n$ to be exponentially large w.r.t. $d'$. If $d'$ is too large, one can try to reduce $d'$ as long as the original data has a lower-dimensional structure (in the sense of singular values $\sigma_i(M)$'s). In the case when further decreasing $d'$ is not possible, the lower bound in  (Boedihardjo et al. 2022)  implied that the rate $n^{-1/d'}$ is optimal.

Reviewer comment (Experiments):
More empirical evaluation would have been nice for this venue. I understand that this is mainly a theoretical work, but given that it might have very practical applications, more experimental work would have made sense.

Author response:
The main focus of our work is on the theoretical side. The experiments can be seen as a proof of concept to overcome the curse of high dimensionality in the Wasserstein accuracy bound. We add more experiments on the basic yet essential statistics of the synthetic data, including mean and covariance.

Reviewer comment (Evaluation):
The empirical accuracy is not significant when $\varepsilon$ is small, even when as small as $1$, although it does better than a direct application of the work from He et al (2023).

Author response:
We agree the empirical accuracy gets worse when $\varepsilon$ is small. We add some explanation for the behavior of low-dimensional algorithm in Section 5. When $\varepsilon$ is small, $(\varepsilon n)^{-1/d'}$ cannot leverage the strength of low-dimensional projection and hence introduce some unnecessary errors. One possible approach is to enlarge the dataset.
 
Also, optimizing the algorithm more practically could potentially increase its performance. However, it is not the main focus of our work in this paper.

Reviewer comment (Experiments):
Where are the empirical comparisons with He et al (2023) shown in the paper?

Author response:
The empirical comparison is in our simulation. The work of He et al. (2023) has no PCA step, and the low-dimensional improvement significantly improves the accuracy and overcomes the curse of high dimensionality.

Reviewer comment (Writing):
At the bottom of page 2, what are $\delta_{X_i}$ and $\delta_{Y_i}$? Should define them somewhere earlier.

Author response:
Thank you for the suggestion. $\delta_{X}$ denote the point mass measure at $X$ and hence $\mu_{\mathcal{X}}$ is the empirical measure for ${\mathcal{X}}$. We will clarify the definition.

Reviewer comment (Experiments):
It is unclear how a practitioner looking to use this approach would know whether their data fits the regime in which the utility theorems hold. In particular, the experiments are very limited and provide little information on the practical utility of the approach for most real-world datasets. For instance, it is very unclear that the inferences about the impact of choosing a smaller d' would generalize to other data.

Author response:
(1) The main focus of our work is on the theoretical side.  We do not run extensive experiments to test the utility of the data. The experiment can be seen as a proof of concept for the Wasserstein accuracy bound we proved. As a result, the low-dimensional algorithm does overcome the curse of high-dimensionality and significantly improves the original algorithm in both accuracy and time and space complexity.

(2) Thank you for the question. We mentioned the *Adaptive and private choices of $d'$* at Section 1.1. We will add a more detailed discussion for the choice of $d'$ in the Appendix E. The main idea for the choice of $d'$ is to balance different terms in the error bound of Theorem 4.2, by finding the best trade-off between low-rank approximation error from SVD and error related to privacy guarantee. Such a minimization problem would be time-efficient as we only need the values of the singular values of the private covariance matrix.

Reviewer comment (Experiments):
The experiments need to be extended to a much wider set of datasets and downstream tasks. Currently, the only evaluation is on the accuracy of a prediction, but one of the main benefits of sharing synthetic data is that in principle it could be used for a large stream of data analysis tasks, beyond training ML models (e.g., descriptive statistics and regression tasks).

Author response:
(2) Indeed, the goal of our algorithm is to ensure the utility of the synthetic data in various down-streaming tasks. We add the comparison of some more basic yet essential statistics of the dataset, including the mean and the covariance, in Appendix~F.

(3) Yes, by our Wasserstein distance bound we showed in Theorem 4.2, one can use the synthetic dataset output from Algorithm 1 for mean and covariance estimation and many other statistical inferences by taking a different Lipschitz test function $f$ in the Kantorovich-Rubinstein Duality expression of Wasserstein distance. The information of Kantorovich-Rubinstein duality of $W_1$ is included in Appendix A.

Reviewer comment (Writing):
The paper would benefit from further intuition and descriptions of the algorithms in the surrounding text to make it more readable. Currently, the paper explains what each part of the algorithm does but lacks explanation of why this is a good idea.

Author response:
(3) Thank you for the suggestion. The main idea of our approach is to generate a private subspace and create private coordinates after projecting data onto that private subspace. We add a non-technical summary of the algorithm in the introduction.