Paper: Differentially Private Principal Component Analysis for Vertically Partitioned Data (ICLR-2024)

Reviewer comment (Experiments):
Although the paper includes experiments on real-world datasets, the number of experiments is relatively small, which could limit the generalizability of the results.

Author response:
**Response.** We believe this is a misunderstanding. First of all, we have included an *extensive* set of experiments, covering different parameter regimes, including both high-dimensional datasets with $n\ll m$ (CiteSeer and Gene), as well as the KDDCUP dataset, which was commonly used for evaluating PCA (e.g., see Chaudhuri et al. [1]), and the recently released ACSIncome dataset [2] that is commonly used for evaluating FL algorithms. In terms of *theory*, we have also proved that our solution is able to achieve comparable performance as the strong centralized baseline.  Both the empirical results and theoretical analysis confirm the performance and generalizability of our solution.

Reviewer comment (Experiments):
Comparisons with other existing solutions for differentially private principal component analysis for vertically partitioned data are not presented in the paper.

Author response:
**Response.** Thanks for raising this concern. As we have mentioned, existing solutions for private PCA for vertically partitioned data that utilize MPC alone do not provide any rigorous DP guarantees (e.g., see [3,4]). Other works, which attempted to achieve DP under vertical FL by letting one of the clients or a third party to inject random DP noises to the sensitive outcome, are **not differentially private** either, as the party might be curious and infer the sensitive outcome (e.g., see [5,6]). As a result, these baselines are **not comparable** with ours, which simultaneously preserves two levels of DP, regarding both a curious server and the curious clients in FL, without assuming any trusted third parties. The only baseline that is comparable to ours is the baseline introduced on page 6 (detailed in Algorithm 3 of page 14), whose performance is much worse than our SPCA, as illustrated in both the theoretical analysis and empirical evaluations.

Reviewer comment (Novelty):
Wang et al. did "Differentially Private Principal Component Analysis Over Horizontally Partitioned Data," but what is your novelty for vertically partitioned data?

Author response:
**Response.** Thanks for the question. We would like to first clarify that existing DP solutions for **horizontally partitioned data** (including the approach by Wang et al. [7]) **do not apply** to our setting of **vertically partitioned data**. The key reason, as we have mentioned in the introduction section, is non-linearity. In the horizontal setting, the clients are able to linearly aggregate their locally perturbed results whereas in the vertical setting, different clients must collaborate to compute the target. In the case of PCA, each entry of the covariance matrix is computed as the inner product between two column vectors possessed by different clients. As a result, the perturbation process cannot be done by a single client alone, since otherwise, the client would learn the sensitive outcome, violating differential privacy.

In terms of contribution, we present the first DP algorithm for PCA over *vertically partitioned data* that simultaneously protects the partitioned data from both the curious server and curious clients, without assuming any trusted third parties. In addition, our algorithm achieves comparable performance as the strong centralized baseline, which is also **a first of its kind** in the vertical FL literature. Although it is well known that DP algorithms under *horizontal FL* are able to achieve comparable performance as in the centralized setting, with the help of secure aggregation or secure shuffling, whether the same can be done under *vertical FL*, however, *has not been answered until this work*. We provide a positive answer in this work, which, we believe is a notable contribution to the privacy community.

Reviewer comment (Reproducibility):
Can you elaborate on how the proposed solution can be practically applied?

Author response:
**Response.** To apply our solution in practice, there are four steps in general, as outlined in Algorithm 1 on page 7.

First, each client independently discretizes her local data partition. This computation can be done efficiently on the client side since only multiplication and random rounding are involved (see Algorithm 2).

Next, each client independently samples Skellam noises on the local side privately, which is done by taking the difference between two identically distributed independent Poisson random variables. Sampling algorithms for Poisson are implemented in commonly used libraries such as NumPy and SciPy.

The clients then collaboratively compute the sum of the covariance matrix and the sampled local Skellam noises (as in Eq.(7) on page 6) using any secure multiparty computation (MPC) algorithm without affecting the privacy guarantees (e.g., using the classic BGW protocol). The computation of Eq.(7) involves only summation and multiplication, which are considered elementary operations and can be done efficiently in modern MPC algorithms (e.g., SPDZ by Keller et al. [8]).

Finally, the server reconstructs the covariance matrix from the outcome of MPC in the previous step and computes the $k$-dimensional singular subspace from the covariance matrix on her side. The server is free to use any centralized algorithm for the computation without affecting the privacy guarantees since post-processing preserves DP (e.g., using the power iteration method for efficient computation of eigenvectors [9]).

Reviewer comment (Experiments):
The paper mentions secret sharing, but then doesn't mention an important baseline to consider. In particular, the brute-force approach which needs "minimal (*)" noise is that the clients secret-share their portion of the data, compute the covariance matrix together in secret shared form (only requiring additions and multiplications, which under SSS can be performed rather efficiently once the data is secret shared), together add the (minimally required) noise, and then reveal the noisy covariance matrix C. This algorithm is clearly differentially private and doesn't leak intermediate results. The only thing another approach can hope to do better is to require fewer communications (and computation). Unless it can be shown that the proposed method is significantly less expensive than this fully secret-sharing-based approach, it does not seem a very valuable contribution.

Author response:
**Response.** Indeed, we have considered the brute-force algorithm you mention above, where for each $C(i,j)$, only the corresponding clients who own columns $i$ and $j$ (say client $p$ and client $q$) participate in the secret sharing process. However, this brute-force approach has the drawback that both client $p$ and client $q$ know exactly *half* the overall DP noise. As a result, it is *easier* for client $q$ to infer the data of client $p$, compared with our solution, where both client $p$ and $q$ know only $1/N$ of the overall DP noise. In cases when $N>2$, our solution is *more private considering curious clients*.

Reviewer comment (Evaluation):
Algorithm 1 appears not to be much more efficient than the brute-force algorithm described above, as it needs to involve all N clients for the computation of each inner product of columns i and j not belonging to the same client. This implies that Algorithm 1 is not significantly faster than the described baseline, and it could have been made faster by computing D[:,i].D[:,j] using a multi-party computation involving only the clients owning columns i and j.

Author response:
Thanks for your kind explanations.

We agree that the brute-force approach you have suggested *indeed is better* than our proposed solution in terms of the privacy-utility trade-off from the perspective of a *curious client*, who no longer has the knowledge of $Sk(\mu/N)$. 

However, we would like to point out that, still, the difference *may be negligible* in scenarios where $N$ is large (e.g., $N=20$). In addition, in certain scenarios, the cost of the brute-force approach could be more significant (despite that it is a constant regarding the size of data). For example, considering the Skellam noise generation with a large $\mu$ (corresponding to strong privacy constraints), the clients would need to collaboratively generate $\mu$ *independent copies* of $Sk(1)$ (let's say $\mu$ is an integer for the ease of discussion), which could be time-consuming. On the other hand, in our solution, each of the $N$ clients independently generates a much smaller Skellam noise in a more efficient offline manner, which could be an advantage under strong privacy constraints and when the number of records is small.

Reviewer comment (Theory):
When the abstract and comments refer to "minimal noise", this is not truly the minimal noise, but rather the smallest amount of noise for which the proof of DP is easy and straightforward. There is no proof that there does not exist an even smaller amount of noise (possibly where not every component C[i,j] gets independent noise) that can also be proven to be DP.

Author response:
**Response.** You are correct. We could not technically prove that our noise is minimal. We have revised the term *minimal noise* to ``noise of scale comparable with the strong centralized baseline'' in the draft.

Reviewer comment (Theory):
The proof of the main result on privacy contains several mistakes, making it hard to verify its overall correctness, even if at a high level the result seems plausible. For example:
* "Since the L2 norm for each row in D and D' is bounded by \(\sqrt{\gamma^2+n}\) ... we have that ... \(\|D^\top D - D'^\top D'\|_F^2 \le \gamma^2 + n\)": This is questionable because the bound on the norm of these inner products would also be expected to be linear in \(\sqrt{m}\), with m being the number of rows.
* "In addition, that the L1 norm of an integer-valued vector v is always less than or equal to \(\|v\|_2^2\) and \(\sqrt{n}\|v\|_2\)": This sentence is not fully grammatically clear, and it is incorrect that the L1 norm of v (\(\sum_i |v_i|\)) is always smaller than \(\|v\|_2^2 = \sum_i v_i^2\), especially for integer-valued v where some components could have absolute values greater than 1.
* The text refers to lemma 1 without first making all its parameters explicit (e.g., \(\Delta_1, \Delta_2, ...\)), which would significantly help in understanding.

Author response:
**Response.** We apologize for the unclear presentation of the proofs. We have revised the proof. In particular, we have included more details to explain that the sensitivity of the covariance matrix of the quantized data is independent of $m$. We also note that since every non-negative integer $u$ is smaller than or equal to $u^2$ (equality holds for $u=0,1$), we have that $||v||_1\leq ||v||_2^2$ for any integer vector $v$. Finally, we have explicitly stated the parameters in the proof for Lemma 2. Please refer to Page 15 of the revised draft for more details.

Reviewer comment (Theory):
Question — It is assumed that what is described in "Baseline in vertical FL" corresponds to what is more commonly known as "Local differential privacy", i.e., every client adds enough noise that the published data does not allow an adversary to reveal any sensitive information.

Author response:
**Response.** You are correct. The vertical FL baseline is indeed a local DP algorithm. As we have mentioned in the draft, other existing vertical FL approaches either involve a client, the server, or a third party to perform the noise injection step, which is not differentially private considering that these parties can be curious or do not provide any rigorous DP analysis (such as [7]).

Reviewer comment (Theory):
The utility result (Lemma 3) doesn't show $N$, which is an important factor in vertical FL. It would be great to show how $N$ influences the results empirically.

Author response:
This is an insightful question. The utility result of Lemma 3 and the empirical performance of SPCA is indeed *independent* of $N$, which is exactly why SPCA achieves *comparable performance* as the strong centralized baseline (think of it as $N=1$). To explain, recall that the core idea of SPCA is to aggregate $N$ independent Skellam noises distributed as $Sk(\mu/N)$ from the client side into $Sk(\mu)$, a larger Skellam noise. Here, the scale of $\mu$, which determines the result utility, is in turn determined by the required server-observed level of DP, which is *independent* of $N$. 

Instead, what $N$ influences is the level of *client-observed* DP (with the level of server-observed DP fixed), since each client knows her local DP noise out of the overall N contributions. As $N$ increases, the client-observed level of DP becomes stronger since each client’s knowledge about the overall noise becomes smaller. As an illustration, in what follows, we fix the server-observed level of DP to $\epsilon=4$, $\delta=10^-5$ and vary $N$ from {10,20,40,80,160,320,640} for the ACSIncome data with $d=817$. 

We report the corresponding levels of *client-observed* $\epsilon$ with $\delta$ fixed to $10^{-5}$ for $\gamma=2^{12}$. 

| client count $N$ | 10 | 20 | 40 | 80 | 160 | 320 | 640 | 
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| **client-observed** $\epsilon$ | 9.61 | 8.97 | 8.68 | 8.55 | 8.48 | 8.45| 8.43 |

Note that the change of $\epsilon$ is almost negligible when $N$ reaches $20$. A similar conclusion also applies to other choices of $d$, $\gamma$, as well as different levels of server-observed $\epsilon$. We have revised the draft to incorporate the above discussion.

Reviewer comment (Evaluation):
It would be meaningful to present the time/communication cost that happened during the MPC, which is dependent on the choice of $\gamma$.

Author response:
Thanks for the suggestion. $\gamma$ represents the level of quantization granularity, which is reflected as the number of bits (namely, $\log \gamma$) in communication. For example, quantizing a real number from $0$ to $1$ with $\gamma=2^{12}$ results in a $12$-bit vector. In the case of the classic BGW protocol, the communication is $O(\log \gamma)$ for sharing the secrets. The time complexity, on the other hand,  is independent of $\gamma$ as the computation for constructing the secret shares (matrix multiplication) and reconstructing the secret (polynomial interpolation) are done on the client side, and the algorithms for these computations are usually dependent on the *size* of the input, rather than the *range* of the input. For example, the Newton interpolation on $N$ points ($N$ secret shares of all clients) incurs a time complexity of $O(N^2)$, which is independent of $\gamma$. In general, the time and communication complexity may vary for different MPC protocols and implementations. We have revised the draft to incorporate the above discussion.

Reviewer comment (Experiments):
Dataset release in RMGM-OLS [1] would provide another reasonable baseline: unlike the baseline in the paper which adds noise to the data matrix directly, it adds noise after a random projection, which can reasonably reduce the scaling of noise.

Author response:
Thanks for the reference. Although random projection is an interesting idea, RMGM-OLS by Wu et al. does not apply to our problem. In particular, the utility goal of Wu et al. (see Eq.(1) on page 3 of their paper) considers linear regression, which is a *supervised learning* problem. Based on linear regression, the clients are able to *train* the model weights to better predict the label of the data, a key component of RMGM-OLS. On the contrary, we consider PCA, an *unsupervised learning* problem where there is no target label. As a result, it is not straightforward to apply RMGM-OLS to PCA and compare it with our solution. We have revised the draft to incorporate the above discussion.