DP-PCA: Statistically Optimal and Differentially Private PCA
Anonymous Author(s)
AffiliationAddress email
Abstract
We study the canonical statistical task of computing the principal component fromn i.i.d.
data in d dimensions under ( ε, δ ) -differential privacy.
Although extensivelystudied in literature, existing solutions fall short on two key aspects: ( i ) even for
Gaussian data, existing private algorithms require the number of samples n toscale super-linearly with d , i.e., n = Ω( d 3 / 2 ) , to obtain non-trivial results whilenon-private PCA requires only n = O ( d ) , and ( ii ) existing techniques suffer froma non-vanishing error even when the randomness in each data point is arbitrarilysmall.
We propose DP-PCA, which is a single-pass algorithm that overcomes bothlimitations.
It is based on a private minibatch gradient ascent method that relieson private mean estimation to add minimal noise required to ensure privacy byadapting to the geometry of a given minibatch of gradients.
For sub-Gaussian data,we provide nearly optimal statistical error rates even for n =
˜
O
( d ) .
Furthermore,we provide a lower bound showing that sub-Gaussian style assumption is necessaryin obtaining the optimal error rate.
Introduction
Principal Component Analysis (PCA) is a fundamental statistical tool with multiple applicationsincluding dimensionality reduction, data visualization and noise reduction.
Naturally, it is a keypart of most standard data analysis and ML pipelines.
However, when applied to data collectedfrom numerous individuals, such as the U S. Census data, outcome of PCA might reveal highlysensitive personal information.
We investigate the design of privacy preserving PCA algorithms andthe involved privacy/utility tradeoffs, for computing the first principal component, that should serveas the building block of more general rank- k PCA.
Differential privacy (DP) is a widely accepted mathematical notion of privacy introduced in [21],which is a standard in releasing the U.S. Census data [2] and also deployed in commercial systems[66, 25, 27].
A query to a database is said to be ( ε, δ ) -differentialy
private if a strong adversarywho knows all other entries but one cannot infer that one entry from the query output, with highconfidence.
The parameters ε and δ restricts the confidence as measured by the Type-I and II errors[44].
Smaller values of ε ∈ [0 , ∞ ) and δ ∈ [0 , 1] imply stronger privacy and plausible deniability forthe participants.
For non-private PCA with n i.i.d.
samples in d dimensions, the popular Oja’s algorithm (provided in Algorithm 1) achieves the optimal error of sin(ˆ v, v 1 ) =
˜Θ( (cid:112) d/n ) , where the error is measured bythe sine function of the angle between the estimate, ˆ v , and the principal component, v 1 , [41].
For differentially private PCA, there is a natural fundamental question: what is the extra cost we pay inthe error rate for ensuring ( ε, δ ) -DP?
We introduce a novel approach we call DP-PCA (Algorithm 3) and show that it achieves an errorbounded by sin(ˆ v, v ) =
˜ O ( (cid:112) d/n + d/ ( εn )) for sub-Gaussian-like data defined in Assumption 1,which is a broad class of light-tailed distributions that includes Gaussian data as a special case.
The second term characterizes the cost of privacy and this is tight; we prove a nearly matchinginformation theoretic lower bound showing that no ( ε, δ ) -DP algorithm can achieve a smaller error.
This significantly improves upon a long line of existing private algorithms for PCA, e.g., [14, 9,36, 34, 23].
These existing algorithms are analyzed for fixed and non-stochastic data and achievesub-optimal error rates of O ( (cid:112) d/n + d 3 / 2 / ( εn )) even in the stochastic setting we consider.
A remaining question is whether the sub-Gaussian-like assumption, namely Assumption A.4, isnecessary or if it is an artifact of our analysis and our algorithm.
It turns out that such an assumptionon the lightness of the tail is critical; we prove an algorithmic independent and information theoreticlower bound (Theorem 5.4) to show that, without such an assumption, the cost of privacy is lowerbounded by Ω( (cid:112) d/ ( εn )) .
This proves a separation of the error depending on the lightness of the tail.
We start with the formal description of the stochastic setting in Section 2 and present Oja’s algorithmfor non-private PCA.
Our first attempt in making this algorithm private in Section 3 already achievesnear-optimal error, if the data is strictly from a Gaussian distribution.
However, there are tworemaining challenges that we describe in detail in Section 4: ( i ) the excessive number of iterations of
Private Oja’s Algorithm (Algorithm 2) prevents using typical values of ε used in practice, and ( ii ) forgeneral sub-Gaussian-like distributions, the error does not vanish even when the noise in the data (asmeasured by a certain fourth moment of a function of the data) vanishes.
The first challenge is due tothe analysis that requires amplification by shuffling [24] that is restrictive.
The second is due to itsreliance on gradient norm clipping [1] that does not adapt to the geometry of the current gradients.
This drives the design of DP-PCA in Section 5 that critically relies on two techniques to overcomeeach challenge, respectively.
First, minibatch SGD (instead of single sample SGD) significantlyreduces the number iterations, thus obviating the need for amplification by shuffling.
Next, privatemean estimation (instead of gradient norm clipping and noise adding) adapts to the geometry of theproblem and adds the minimal noise necessary to achieve privacy.
The main idea of this geometryadaptive stochastic gradient update is explained in detail in Section 6, along with a sketch of a proof.
Notations.
For a vector x ∈ R d , we use ∥ x ∥ to denote the Euclidean norm.
For a matrix X ∈ R d × d ,we use ∥ X ∥ 2 = max ∥ v ∥ =1 ∥ Xv ∥ 2 to denote the spectral norm.
We use I d to denote d × d identitymatrix.
For n ∈ Z + , let [ n ] := { 1 , 2 , . . . , n } .
Let S d − 1denote the unit d -sphere of ℓ 2 , i.e., 65 S d − :=
{ x ∈ R d : ∥ x ∥ = 1 } .
˜ O
() hides logarithmic factors in n , d , and the failure probability ζ .
Problem formulation and background on DP
Typical PCA assumes i.i.d.
data { x i ∈ R d } from a distribution and finds the first eigenvector ofΣ =
E [( x i − E
[ x i ])
( x i − E
[ x i ])
⊤ ] ∈ R d × d .
Our approach allows for a more general class of data{ A i ∈ R d × d } that recovers the standard case when A i =
( x i − E
[ x i ])
( x i − E
[ x i ]) ⊤ .
Assumption 1 ( (Σ , { λ i } di =1 , M, V, K, κ, a, γ 2 ) -model) .
Let A 1 , A 2 , . . .
, A n ∈ R d × d be a sequenceof (not necessarily symmetric) matrices sampled independently from the same distribution thatsatisfy the following with PSD matrices Σ ∈ R d × d and H u ∈ R d × d , and positive scalar parameters
M, V, K , κ , a , and γ 2 :
A.1.
Let Σ :=
E
[ A i ] , for a symmetric positive semidefinite (PSD) matrix Σ ∈ R d × d , λ i denote thei -th largest eigenvalue of Σ , and κ :=
λ 1 / ( λ 1 − λ 2 ) ,
A.2.
∥
A i − Σ ∥ 2 ≤ λ 1 M almost surely,
A.3. max
[Equation]
( A i [Equation] Σ
[Equation]
A i
[Equation]
⊤
[Equation]
2 , (cid:13)(cid:13)
[Equation]
(cid:2)
( A i
[Equation]
Σ) ⊤ ( A i
[Equation]
Σ) (cid:3)(cid:13)(cid:13) 2 (cid:9) ≤ λ 21 V ,
A.4.
max
[Equation] [Equation]
E
[Equation] [Equation] 2 a ) (cid:19)(cid:21) ≤ 1 , where H u := (1 /λ 21 ) E
[( A i −Σ) uu ⊤ ( A i − Σ) ⊤ ] .
We denote γ 2 := max ∥ u
∥ =1 ∥
H u ∥ 2 .
The first three assumptions are required for PCA even if privacy is not needed.
The last assumptionprovides a Gaussian-like tail bound that determines how much noise we need to introduce in thealgorithm for ( ε, δ ) -DP.
The following lemma is useful in the analyses.
Lemma 2.1.
Under A.1 and A.4 in Assumption 1, for any unit vector u , v , with probability 1 − ζ ,
[Equation]
2.1 Oja’s algorithm
In a non-private setting, the following streaming algorithm introduced in [63] achieves optimal samplecomplexity as analyzed in [41].
It is a projected stochastic gradient ascent on the objective defined onthe empirical covariance: max ∥ w ∥ =1 (1 /n ) (cid:80) ni
=1 w ⊤ A i w .
Algorithm
Central to our analysis is the following error bound on Oja’s Algorithm from [41].
Theorem 2.2 ([41, Theorem 4.1]) .
Under Assumptions A.1-A.3, suppose the step size η t =α ( λ 1 − λ 2 )( ξ + t ) for some α > 1 / 2 and ξ
[Equation]
If T > ξ then there exists a constant C > 0 such that Algorithm 1 outputs w T achieving w.p. 1 − ζ ,
[Equation]
2.2 Background on Differential Privacy
Differential privacy (DP), introduced in [21], is a de facto mathematical measure for privacy leakageof a database accessed via queries.
It ensures that even an adversary who knows all other entriescannot identify with a high confidence whether a person of interest participated in a database or not.
Definition 2.3 (Differential privacy [21]) .
Given two multisets S and S ′ , we say the pair ( S, S ′ )is neighboring if | S \ S ′ | + | S ′ \ S | ≤ 1 .
We say a stochastic query q over a dataset S satisfies( ε, δ ) -differential
privacy for some ε > 0 and δ ∈ (0 , 1) if P ( q ( S ) ∈
A )
≤ e ε P ( q ( S ′ )
∈ A ) + δ forall
neighboring ( S, S ′ ) and all subset A of the range of q .
Small values of ε and δ ensures that the adversary cannot identify any single data point with highconfidence, thus providing plausible deniability.
We provide useful DP lemmas in Appendix B. Withinour stochastic gradient descent approach to PCA, we rely on the Gaussian mechanism to privatizeeach update.
The sensitivity of a query q is defined as ∆ q :=
sup neighboring ( S,S ′ )
∥ q ( S ) − q
S ′ )
∥ .
Lemma 2.4 (Gaussian mechanism [22]) .
For a query q with sensitivity ∆ q , ε ∈ (0 , 1) , and δ ∈ (0 , 1) ,the Gaussian mechanism outputs q ( S ) + N (0 , (∆ q ( (cid:112) 2 log(1 .
25 /δ ))
/ε )
2 I d ) and achieves ( ε, δ ) -DP.
Another mechanism we frequently use is the private histogram learner of [51], whose analysis isprovide in Appendix B, along with various composition theorems to provide end-to-end guarantees.
2.3 Comparisons with existing results in private PCA
We briefly discuss the most closely related work and provide more previous work in Appendix A. Mostexisting results assume a fixed data under a deterministic setting where each sample has a boundednorm, ∥ x i ∥ ≤ β , and the goal is to find the top eigenvector of ˆΣ :=
(1 /n ) (cid:80) ni
=1 ( x i − ˆ µ )( x i − ˆ µ ) ⊤
Algorithm
Input: S =
{ A i ∈ R d × d } ni =1 , privacy ( ε, δ ) , learning rates { η t } nt = 1 Randomly permute S and choose w 0 uniformly at random from the unit sphere 2 Set DP noise multiplier: α ← C ′ log( n/δ ) / ( ε √ n ) 3 Set clipping threshold:
β ← Cλ 1 √ d ( Kγ log a ( nd/ζ ) + 1) 4 for t=1, 2, ..., n do
Sample z t ∼ N (0 , I d ) w ′ t ← w t − 1 + η t clip β ( A t w t − 1 ) + 2 η t βαz t where clip β ( x ) = x · min { 1 , β ∥ x ∥ 2 } w t ← w ′ t / ∥ w ′ t ∥ 8 Return w n for the empirical mean ˆ µ . For the purpose of comparisons, consider Gaussian x i ∼ N (0 , Σ) with∥ x i ∥ ≤ β = O ( (cid:112) λ 1 d log( n/ζ )) for all i ∈ [ n ] with probability 1 − ζ .
The first line of approaches in[9, 14, 23] is a Gaussian mechanism that outputs PCA( (cid:98) Σ + Z ) , where Z is a symmetric matrix withi.i.d.
Gaussian entries with a variance (( β 2 /nε )
(cid:112) 2 log(1 .
25 /δ )) 2 to ensure ( ε, δ ) -DP.
The tightestresult in [23, Theorem 7] achieves
[Equation]
with high probability, under a strong assumption that the spectral gap is very large: λ 1 − λ 2 =ω ( d 3 / 2 (cid:112) log(1 /δ ) / ( εn )) .
In a typical scenario with λ 1 =
O
(1) , this requires a large sample sizeof n = ω ( d 3 / 2 /ε ) .
Since this Gaussian mechanism does not exploit the statistical properties ofi.i.d.
samples, the second term in this upper bound is larger by a factor of d 1 / 2 compared to theproposed DP-PCA (Corollary 5.2).
The error rate of Eq.
(3) is also achieved in [36, 34] by adding
Gaussian noise to the standard power method for computing the principal components.
When thespectral gap, λ 1 − λ 2 , is smaller, it is possible to trade-off the dependence in κ and the sampling ratiod/n , which we do not address in this work but is surveyed in Appendix A.
First attempt: making Oja’s Algorithm private
Following the standard recipe in training with DP-SGD, e.g., [1], we introduce Private Oja’s Algorithmin Algorithm 2.
At each gradient update, we first apply gradient norm clipping to limit the contributionof a single data point and next add an appropriately chosen Gaussian noise from Lemma 2.4 to achieve( ε, δ ) -DP, end-to-end.
The choice of clipping threshold β ensures that, with high probability underour assumption, we do not clip any gradients.
The choice of noise multiplier α ensures ( ε, δ ) -DP.
One caveat in streaming algorithms is that we access data n times, each with a private mechanism,but accessing only a single data point at a time.
To prevent excessive privacy loss due to such a largenumber of data accesses, we apply a random shuffling in line 1 Algorithm 2, in order to benefit froma standard amplification by shuffling [24, 29].
This gives an amplified privacy guarantee that allowsus to add a small noise proportional to α
= O (log( n/δ ) / ( ε √ n )) .
Without the shuffle amplification,we will instead need a larger noise scaling as α = O (log( n/δ ) /ε ) , resulting in a suboptimal utilityguarantee.
However, this comes with a restriction that the amplification holds only for small valuesof ε
= O ( (cid:112) log( n/δ ) /n ) .
Our first contribution in the proposed DP-PCA (Algorithm 3) is to expandthis range to ε
= O (1) , which includes the practical regime of interest ε ∈ [1 / 2 , 5] .
Lemma 3.1 (Privacy) .
If ε =
O ( (cid:112) log( n/δ ) /n ) and the noise multiplier is chosen to be α =Ω (log( n/δ ) / ( ε √ n )) , then Algorithm 2 is ( ε, δ ) -DP.
Under Assumption 1, we select gradient norm clipping threshold β such that no gradient exceeds β .
Lemma 3.2 (Gradient clipping) .
Let β
= Cλ 1 √ d ( Kγ log a ( nd/ζ ) + 1) for some constant C > 0 .
Then with probability 1 − ζ , ∥
A t w t − 1 ∥ ≤ β for any fixed w t − 1 independent of A t , for all t ∈ [ n ] .
We provide proofs of both lemmas and the next theorem in Appendix D. When no clipping is applied,we can use the standard analysis of Oja’s Algorithm from [41] to prove the following utility guarantee.
Theorem 3.3 (Utility) .
Given n i.i.d.
samples { A i ∈ R d × d } ni =1 satisfying Assumption 1 withparameters (Σ , M, V, K, κ, a, γ 2 ) , if
[Equation]
with a large enough constant, then there exists a positive universal constant c 1 and a choice oflearning rate η t that depends on ( t, M , V , K , a , λ 1 , λ 1 − λ 2 , n , d , ε , δ ) such that Algorithm 2 witha choice of ζ = 0 .
01 outputs w n that achieves with probability 0 . 99 ,
[Equation]
where (cid:101) O ( · ) hides poly-logarithmic factors in n , d , 1 /ε , and log(1 /δ ) and polynomial factors in K .
The first term in Eq.
(5) matches the non-private error rate for Oja’s algorithm in Eq.
(2) withα =
O (log n ) and T = n , and the second term is the price we pay for ensuring ( ε, δ ) -DP.
Remark 3.4.
For a canonical setting of a Gaussian data with A i
= x i x
⊤ i and x i ∼ N (0 , Σ) , wehave, for example from [64, Lemma 1.12], that M
= O ( d log( n )) , V = O ( d ) , K = 4 , a = 1 , andγ 2 =
O (1) .
Theorem 3.3 implies the following error rate:
[Equation]
Comparing to a lower bound in Theorem 5.3, this is already near optimal.
However, for generaldistributions satisfying Assumption 1, Algorithm 2 (in particular the second term in Eq.
canbe significantly sub-optimal.
We explain this second weakness of Private Oja’s Algorithm in thefollowing section (the first weakness is the restriction on ε
=
O ( (cid:112) log( n/δ ) /n ) ).
Two remaining challenges
We explain the two remaining challenges in Private Oja’s Algorithm and propose techniques toovercomes these challenges that lead to the design of DP-PCA (Algorithm 3).
First challenge: restricted range of ε = O ( (cid:112) log( n/δ ) /n ) .
This is due to the large number, n , ofiterations that necessitates the use of the amplification by shuffling in Theorem D.1.
We reduce thenumber of iterations with minibatch SGD.
For T
=
O (log 2 n ) and t = 1 , 2 , . .
, T , we repeat
[Equation]
where z t ∼ N (0 , I d ) and the minibatch size is B =
⌊ n/T ⌋ .
Since the dataset is accessed only
T =
O (log 2 n ) times, the end-to-end privacy is analyzed with the serial composition (Lemma B.3)instead of the amplification by shuffling.
This ensures ( ε, δ ) -DP for any ε =
O (1) , resolving the firstchallenge, and still achieves the utility guarantee of Eq.
Second challenge: excessive noise for privacy.
This is best explained with an example.
Example 4.1 (Signal and noise separation) .
Consider a setting with A i
= x
i x ⊤ i and x i
= s i + n iwhere s i
= v with probability half
and s i = − v otherwise for a unit norm vector v and n i ∼
N (0 , σ 2 I ) .
We want to find the principal component of Σ = E
[ x i x ⊤ i ] =
vv ⊤ + σ
I , which is v .
This construction decomposes the signal and the noise.
For A i = vv ⊤
[Equation]
s i n ⊤ i + n i s ⊤ i + n i n ⊤ i , thesignal component is determined by vv ⊤ that is deterministic due to the sign cancelling.
The noisecomponent is s i n ⊤ i + n i s ⊤ i + n i n ⊤ i which is random.
We can control the Signal-to-Noise Ratio(SNR), 1 /σ 2 , by changing σ 2 , and we are particularly interested in the regime where σ 2 is small.
Aswe are interested in σ 2 < 1 , this satisfies Assumption 1 with λ 1 = 1 + σ 2 , λ 2 = σ 2 , V =
O ( dσ 2 ) ,
K
= O (1) , a = 1 , and γ 2 = σ 2 .
Substituting this into Eq.
(5) , Private Oja’s Algorithm achieves
[Equation]
where we are interested in σ 2 < 1 .
Input: S =
{ A i } ni =1 , ( ε, δ )
, batch size B ∈ Z
+ , learning rates { η t } ⌊ n/B ⌋ t = , probability ζ ∈ (0 , 1)
Choose w 0 uniformly at random from the unit sphere 2 for t = 1 , 2 , . .
, T =
⌊ n/B ⌋ do
Run Private Top Eigenvalue Estimation (Algorithm 4) with ( ε/ 2 , δ/ 2) -DP and failure probability ζ/ (2 T ) on { A B ( t − 1)+ i w t − 1 } ⌊ B/ 2 ⌋ i =1 .
Let the returned estimation be ˆΛ t > 0 .
Run Private Mean Estimation (Algorithm 5) with ( ε/ 2 , δ/ 2) -DP, failure probability ζ/ (2 T ) , and the estimated eigenvalue 2ˆΛ t on (cid:8) A B ( t − 1)+ ⌊ B/ 2 ⌋ + i w t − 1 (cid:9) i ∈⌊ B/ 2 ⌋ .
Let the returned mean gradient estimate be ˆ g t ∈ R d .
w
[Equation]
t ←
[Equation]
+
[Equation]
, w
[Equation]
∥ 6 Return w T
This is problematic since the second term, due to the DP noise, does not vanish as the randomnessσ 2 in the data decreases.
We do not observe this for Gaussian data where signal and noise scaleproportionally as shown below.
We reduce the noise we add for privacy, by switching from a simplenorm clipping, that adds noise proportional to the norm of the gradients, to private estimation, thatonly requires the noise to scale as the range of the gradients, i.e. the maximum distance betweentwo gradients in the minibatch.
The toy example above showcases that the range can be arbitrarilysmaller than the maximum norm (Fig. 1).
We want to emphasize that although the idea of usingprivate estimation within an optimization has been conceptually proposed in abstract settings, e.g., in[46], DP-PCA is the first setting where ( i ) such separation between the norm and the range of thegradients holds under any statistical model, and hence ( ii ) the long line of recent advances in privateestimation provides significant gain over the simple DP-SGD [1].
Differentially Private Principal Component Analysis (DP-PCA)
Combining the two ideas of minibatch SGD and private mean estimation, we propose DP-SGD.
Weuse minibatch SGD of minibatch size B =
O ( n/ log 2 n ) to allow for larger range of ε
= O (1) .
Weuse Private Mean Estimation to add an appropriate level of noise chosen adaptively according to
Private Eigenvalue Estimation.
We describe details of both sub-routines in Section 6.
We show an upper bound on the error achieved by DP-PCA under an appropriate choice of thelearning rate.
We provide a complete proof in Appendix E.1 that includes the explicit choice of thelearning rate η t in Eq.
(67), and a proof sketch is provided in Section 6.1.
Theorem 5.1.
For ε ∈ (0 , 0 . 9) , DP-PCA guarantees ( ε, δ ) -DP for all S , B , ζ , and δ .
Given ni.i.d.
samples { A i ∈ R d × d } ni =1 satisfying Assumption 1 with parameters (Σ , M, V, K, κ, a, γ 2 ) , ifn
[Equation]
with a large enough constant and δ ≤ 1 /n , then there exists a positive universal constant c 1 anda choice of learning rate η t that depends on ( t, M , V , K , a , λ 1 , λ 1 − λ 2 , n , d , ε , δ ) such that
T =
⌊ n/B ⌋ steps of DP-PCA in Algorithm 3 with choices of ζ = 0 .
01 and B = c 1 n/ (log n ) 2 , outputs w T such that with probability 0 .
[Equation]
where (cid:101) O ( · ) hides poly-logarithmic factors in n , d , 1 /ε , and log(1 /δ ) and polynomial factors in K .
We further interpret this analysis and show that ( i ) DP-PCA is nearly optimal when the data isfrom a Gaussian distribution by comparing against a lower bound (Theorem 5.3); and ( ii ) DPPCA significantly improves upon the private Oja’s algorithm under Example 4.1.
We discuss thenecessity of some of the assumptions at the end of this section, including how to agnostically find theappropriate learning rate scheduling.
Near-optimality of DP-PCA under Gaussian distributions.
Consider the case of i.i.d.
samples{ x i } ni =1 from a Gaussian distribution from Remark 3.4.
Corollary 5.2 (Upper bound; Gaussian distribution) .
Under the hypotheses of Theorem 5.1 and{ A i = x
i x ⊤ i } ni =1 with Gaussian random vectors x i ’s, after T = n/B steps, DP-PCA outputs w Tthat achieves, with probability 0 .
[Equation] [Equation]
We prove a nearly matching lower bound, up to factors of (cid:112) λ 1 /λ 2 and (cid:112) log(1 /δ ) .
One caveat isthat the lower bound assumes pure -DP with δ = 0 .
We do not yet have a lower bound technique forapproximate DP that is tight, and all known approximate DP lower bounds have gaps to achievableupper bounds in its dependence in log(1 /δ ) , e.g., [5, 57].
We provide a proof in Appendix C.1.
Theorem 5.3 (Lower bound; Gaussian distribution) .
Let M ε be a class of ( ε, 0)
-DP estimators thatmap n i.i.d.
samples to an estimate ˆ v ∈ R d .
A set of Gaussian distributions with ( λ 1 , λ 2 ) as the firstand second eigenvalues of the covariance matrix is denoted by P ( λ 1 ,λ 2 ) .
For d>c where c > 0 issome absolute constant, there exists a universal constant C > 0 such that
[Equation]
Comparisons with private Oja’s algorithm.
We demonstrate that DP-PCA can significantlyimprove upon Private Oja’s Algorithm with Example 4.1, where DP-PCA achieves an error boundof sin( w T , v 1 ) =
˜ O (cid:0) σ (cid:112) d/n + σd (cid:112) log(1 /δ ) / ( εn ) (cid:1) .
As the noise power σ 2 decreases DP-PCAachieves a vanishing error, whereas Private Oja’s Algorithm has a non-vanishing error in Eq.
This follows from the fact that the second term in the error bound in Eq.
(10) scales as γ , which canbe made arbitrarily smaller than the second term in Eq.
(5) that scales as ( γ + 1) .
Further, the errorbound for DP-PCA holds for any ε =
O (1) , whereas Private Oja’s Algorithm requires significantlysmaller ε
= O ( (cid:112) log( n/δ ) /n ) .
Remarks on the assumptions of Theorem 5.1.
We have an exponential dependence of the samplecomplexity in the spectral gap, n ≥ exp( κ 2 ) .
This ensures we have a large enough T =
⌊ n/B ⌋ toreduce the non-dominant second term in Eq.
(2), in balancing the learning rate η t and T (which isexplicitly shown in Eqs.
69 and (70) in the Appendix).
It is possible to get rid of this exponentialdependence at the cost of an extra term of ˜ O ( κ 4 γ 2 d 2 log(1 /δ ) /
( εn ) 2 ) in the error rate in Eq.
(10),by selecting a slightly larger T = cκ 2 log 2 n .
A Gaussian-like tail bound in Assumption A.4 isnecessary to get the desired upper bound scaling as ˜ O
( d (cid:112) log(1 /δ )
/ ( εn )) in Eq.
11, for example.
The next lower bound shows that without such assumptions on the tail, the error due to privacy scalesas Ω( (cid:112) d
∧ log(1 /δ )
/ ( εn )) .
We believe that the dependence in δ is loose, and it might be possible toget a tighter lower bound using [47].
We provide a proof and other lower bounds in Appendix C.
Theorem 5.4 (Lower bound without Assumption A.4) .
Let M ε be a class of ( ε, δ ) -DP estimatorsthat map n i.i.d.
samples to an estimate ˆ v ∈ R d .
A set of distributions satisfying Assumptions A.1–A.3with M =
˜ O
( d + (cid:112) nε/d ) , V = O ( d ) and γ =
O (1) is denoted by ˜ P .
For d ≥ 2 , there exists auniversal constant C > 0 such that
[Equation]
Currently, DP-PCA requires choices of the learning rates, η t , that depend on possibly unknownquantities.
Since we can privately evaluate the quality of our solution, one can instead run multipleinstances of DP-PCA with varying η t =
c 1 / ( c 2 + t ) and find the best choice of c 1 > 0 and c 2 > 0 .
Let w T ( c 1 , c 2 ) denote the resulting solution for one instance of { η t =
c 1 / ( c 2 + t ) }
Tt =1 .
We firstset a target error ζ .
For each round i = 1 , . . .
, we will run algorithm for ( c 1 , c 2 ) =
[2 i − 1 , 2 − i +1 ] ×[2 − i +1 , 2 − i +2 . .
, 2 i − 1 ] and ( c 1 , c 2 ) =
[2 − i +1 , 2 − i +2 . . .
, 2 i − 1 ] ×
[2 i − 1 , 2 − i +1 ] , and compute eachsin( w T ( c 1 , c 2 ) , v 1 ) privately, each with privacy budget ε i =
[Equation]
Weterminate the algorithm once there there is a w T ( c 1 , c 2 ) satisfies sin( w T ( c 1 , c 2 ) , v 1 ) ≤ ζ .
It is clearthat
this search meta-algorithm terminate in logarithmic round, and the total sample complexity onlyblows up by a poly-log factor.
Private mean estimation for the minibatch stochastic gradients
DP-PCA critically relies on private mean estimation to reduce variance of the noise required toachieve ( ε, δ ) -DP.
We follow a common recipe from [51, 45, 49, 8, 15].
First, we privately find anapproximate range of the gradients in the minibatch (Alg. 4).
Next, we apply the Gaussian mechanismto the truncated gradients where the truncation is tailored to the estimated range (Alg. 5).
Step 1: estimating the range.
We need to find an approximate range of the minibatch of gradients inorder to adaptively truncate the gradients and bound the sensitivity.
Inspired by a private preconditioning mechanism designed for mean estimation with unknown covariance from [48], we propose to useprivately estimated top eigenvalue of the covariance matrix of the gradients.
For details on the versionof the histogram learner we use in Alg. 4 in Appendix E.2, we refer to [56, Lemma D.1].
Unlike theprivate preconditioning of [48] that estimates all eigenvalues and requires n = (cid:101)
O ( d 3 / 2 log(1 /δ ) /ε )
samples, we only require the top eigenvalue and hence the next theorem shows that we only needn = (cid:101)
O ( d log(1 /δ ) /ε ) .
Theorem 6.1.
Algorithm 4 is ( ε, δ ) -DP.
Let g i
=
A i u for some fixed vector u , where A i satisfies A.1and A.4 in Assumption 1 such that the mean is E [ g i ] =
Σ u and the covariance is E [( g i − Σ u )( g i −Σ u ) ⊤ ] =
λ 21 H u .
With a large enough sample size scaling as
[Equation]
O [Equation] Algorithm 4 outputs ˆΛ achieving ˆΛ ∈ (cid:2) (1 / √ 2) λ 21 ∥ H u ∥ 2 , √ 2 λ 21 ∥
H u ∥ 2 (cid:3) with probability 1 − ζ , wherethe pair ( K > 0 , a > 0) parametrizes the tail of the distribution in A.4 and ˜ O ( · ) hides logarithmicfactors in B, d, 1 /ζ, log(1 /δ ) , and ε .
We provide a proof in Appendix E.2.
There are other ways to privately estimate the range.
Someapproaches require known bounds such as σ 2min ≤ λ 21 ( H u ) ii ≤ σ 2max for all i ∈
[ d ] [51], and otheragnostic approaches are more involved such as instance optimal universal estimators of [17].
Step 2: Gaussian mechanism for mean estimation.
Once we have a good estimate of the topeigenvalue from the previous section, we use it to select the bin size of the private histogram andcompute the truncated empirical mean.
Since truncated empirical mean has a bounded sensitivity, wecan use Gaussian mechanism to achieve DP.
The algorithm is now standard in DP mean estimation,e.g., [51, 45].
However, the analysis is slightly different since our assumptions on g
i ’s are different.
For completeness, we provide the Algorithm 5 in Appendix E.3.
The next lemma shows that the Private Mean Estimation is ( ε, δ ) -DP, and with high probabilityclipping does not apply to any of the gradients.
The returned private mean, therefore, is distributed asa spherical Gaussian centered at the empirical mean of the gradients.
This result requires that we havea good estimate of the top eigenvalue from Alg.
4 such that ˆΛ ≃ λ 21 ∥ H u ∥ 2 .
This analysis impliesthat we get an unbiased estimate of the gradient mean (which is critical in the analysis) with noisescaling as ˜ O
( λ 1 γ (cid:112)
d log(1 /δ ) / ( εB )) , where γ 2 = max u : ∥ u ∥ =1 ∥
H u ∥ 2 (which is critical in gettingthe tight sample complexity in the second term of the final utility guarantee in Eq.
We providea proof in Appendix E.3.
Lemma 6.2.
For ε ∈ (0 , 0 . 9) and any δ ∈ (0 , 1) , Algorithm 5 is ( ε, δ ) -DP.
Let g i
=
A i u for somefixed vector u , where A i satisfies A.1 and A.4 in Assumption 1 such that the mean is E [ g i ] =
Σ uand the covariance is E [( g i − Σ u )( g i − Σ u ) ⊤ ] = λ 21 H u .
If ˆΛ ∈ [ λ 21 ∥
H u ∥ 2 / √ 2 , √ 2 λ 21 ∥
H u ∥ 2 ] ,δ ≤ 1 /B , and B = Ω(( (cid:112)
d log(1 /δ )
/ε ) log( d/ ( ζδ )))
then, with probability 1 − ζ , g i ∈
¯ g +(cid:104) − 3 K (cid:112) ˆΛ log a ( Bd/ζ ) , 3 K (cid:112) ˆΛ log a ( Bd/ζ ) (cid:105) d for all i ∈
[ B ] .
6.1 Proof sketch of Theorem 5.
We choose B = Θ( n/ log 2 n ) such that we access the dataset only T = Θ(log 2 n ) times.
Hence wedo not need to rely on amplification by shuffling.
To add Gaussian noise that scales as the standarddeviation of the gradients in each minibatch (as opposed to potentially excessively large mean of thegradients), DP-PCA adopts techniques from recent advances in private mean estimation.
Namely,we first get a private and accurate estimate of the range from Theorem 6.1.
Using this estimate, ˆΛ ,
Private Mean Estimation returns an unbiased estimate of the empirical mean of the gradients, as longas no truncation has been applied as ensured by Lemma 6.2.
This gives
[Equation]
for z t ∼ N (0 , I ) and β t = 8 K
[Equation]
Using rotation invariance of spherical
Gaussian random vectors and the fact that ∥ w t − 1 ∥ = 1 , we can reformulate it as
[Equation]
This process can be analyzed with Theorem 2.2 with ˜ A t substituting A t .
Discussion
Under the canonical task of computing the principal component from i.i.d.
samples, we show thefirst result achieving an optimal error rate.
This critically relies on two ideas: minibatch SGD andprivate mean estimation.
In particular, private mean estimation plays a critical role in the case whenthe range of the gradients is significantly smaller than the norm; we achieve an optimal error rate thatcannot be achieved with the standard recipe of gradient clipping.
Assumption A.4 can be relaxed to heavy-tail bounds with bounded k -th moment on A i , in which casewe expect the second term in Eq.
(10) to scale as O ( d ( (cid:112) log(1 /δ )
/εn )
1 − 1 /k ) , drawing analogy froma similar trend in a computationally inefficient DP-PCA without spectral gap [57, Corollary 6.10].
When a fraction of data is corrupted, recent advances in [77, 53, 42] provide optimal algorithms for
PCA.
However, existing approach of [57] for robust and private PCA is computationally intractable.
Borrowing ideas from robust and private mean estimation in [56], one can design an efficientalgorithm, but at the cost of sub-optimal sample complexity.
It is an interesting direction to design anoptimal and robust version of DP-PCA.
Our lower bounds are loose in its dependence in log(1 /δ ) .
Recently, a promising lower bound technique has been introduced in [47] that might close this gap.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] , see Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work is theoretical and does not have a direct negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A] We don’t have experiments
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We didn’t use
existing assets.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]