Original: Differentially Private Covariance Revisited
Revised: Differentially Private Covariance Revisited

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In this paper we present three new error bounds, in terms of the Frobenius norm,for covariance estimation under  differential privacy: (1) a worst-case bound of˜ O p d 1 { 4{ ?  n q , which improves the standard Gaussian mechanism ˜
Revised: In this paper, we present two new algorithms for covariance estimation under concentrated differential privacy (zCDP). The first algorithm achieves a Frobenius error of ˜ O p d 1 { 4 ? tr {? n ` ? d { n q , where tr is the trace of the covariance matrix.

Revised: By taking tr “ 1 , this also implies a worst-case error bound of ˜

Original: O p d  { n q for theregime d ą r Ω p n 2 { 3 q ; (2) a trace-sensitive bound that improves the state of the artby a ?
Revised: O p d 1 { 4 {? n q , which improves the standard Gaussian mechanism’s ˜ O p d { n q for the regime d ą r Ω p n 2 { 3 q .

Original: d -factor, and (3) a tail-sensitive bound that gives a more instance-specificresult.
Revised: Our second algorithm offers a tail-sensitive bound that could be much better on skewed data.

Original: The corresponding algorithms are also simple and efficient.
Revised: The corresponding algorithms are also simple and efficient.

Original: Experimentalresults show that they offer significant improvements over prior work.
Revised: Experimental results show that they offer significant improvements over prior work.

Original:  Introduction
Revised:  Introduction

Original: Consider a dataset represented by a matrix X P R d ˆ n , where each column X i , i “ 1 , . .
Revised: Consider a dataset represented by a matrix X P R d ˆ n , where each column X i , i “ 1 , . .

Original:  , ncorresponds to one individual’s information.
Revised: . , n corresponds to an individual’s information.

Original: As standard in the literature, we assume that all the X i ’slive in B d , the d -dimensional ℓ 2 -unit ball centered at the origin.
Revised: As standard in the literature, we assume that all the X i ’s live in B d , the d -dimensional ℓ 2 -unit ball centered at the origin.

Original: In this paper, we revisit the problemof estimating the (empirical) covariance matrix Σ
Revised: In this paper, we revisit the problem of estimating the (empirical) covariance matrix Σ

Original: p X q : “ 1 n ř i
Revised: p X q : “ 1 n ř i

Original: X
Revised: X

Original: i X Ti “ 1 n XX T under differentialprivacy (DP), a fundamental problem  that requires little motivation.
Revised: i X Ti “ 1 n XX T under differential privacy (DP), a fundamental problem in high-dimensional data analytics and machine learning that requires little motivation.

Revised: We often write Σ p X q as Σ when the context is clear.

Original: As with most prior work, weuse the Frobenius norm } r Σ ´ Σ } F to measure the error of the estimated covariance r Σ .
Revised: As with most prior work, we use the Frobenius norm } r Σ ´ Σ } F to measure the error of the estimated covariance r Σ .

Original: We presentimproved and new results for the following three settings with increasing generality.

Original: To better focus,in the main paper weonly state  results under concentrated different privacy (zCDP) [9]; extensions ofour results to pure-DP are given in Appendix G.
Revised: To better focus, in the introduction we state all results under concentrated different privacy (zCDP) [10]; extensions of our results to pure-DP are given in Appendix I.

Original: In cases where prior results do not adopt zCDP, westate their results under approximate-DP, a privacy guarantee that is weaker than zCDP.

Original: The formaldefinitions of these privacy guarantees, as well as their relationships, are given in Section 3.

Original: 1.1 Worst-case Bounds

Original: Because the ℓ 2 -sensitivity of Σ , i.e., max X „ X 1 } Σ p X q ´ Σ p X 1 q} F where X „ X 1 denotes twoneighboring datasets that differ by one column, is O p 1 { n q , the standard Gaussian mechanism [15]achieves an error of ˜ O p d { n q 1 by adding an independent Gaussian noise of scale ˜ O p 1 { n q to each of thed 2 entries of Σ .

Original: In this paper, we show how to achieve ˜ O p d 1 { 4 {? n q , which is better than the Gaussianmechanism when d ą r Ω p n 2 { 3 q .

Original: Note that the hardness of covariance estimation lies between ddimensional mean estimation (only estimating the diagonal entries of Σ ) and d 2 -dimensional meanestimation (treating Σ as a d 2 -dimensional vector).

Original: While the Gaussian mechanism simply equates itwith the latter, our result indicates that the problem is strictly easier, due to the correlations of thed 2 entries of Σ , at least in the high-dimensional regime d ą r Ω p n 2 { 3 q .

Original: Under pure-DP, there is alower bound of r Ω p d { n q [17], which is also conjectured to hold under zCDP/approximate-DP [1].

Original: Ourresult thus refutes this conjecture for d ą r Ω p n 2 { 3 q , while demonstrating a separation of zCDP andpure-DP for covariance estimation.

Original: Lastly, another interesting consequence is that our error bound has utility for d up to ˜

Original: O p n 2 q (utility is lost when the error is ˜ O p 1 q , as returning a zero matrix canalready achieve this error).

Original: This is the highest d that allows for any utility, since even mean estimationrequires d ă ˜ O p n 2 q to have utility under zCDP [17, 9].

Original: 1.2 Trace-sensitive Bounds
Revised: 1.1 A Trace-sensitive Algorithm

Original: For any symmetric matrix A , we use P r A s and Λ r A s to denote its matrices of eigenvectors andeigenvalues, respectively, such that A “ P r A s Λ r A s P r A s T ; we use λ i
Revised: For any symmetric matrix A , we use P r A s and Λ r A s to denote its matrices of eigenvectors and eigenvalues, respectively, such that A “ P r A s Λ r A s P r A s T ; we use λ i

Original: r A s to denote its i th largesteigenvalue.
Revised: r A s to denote its i th largest eigenvalue.

Original: When A “ Σ  , we simply write P “ P r Σ s , Λ “ Λ r Σ s , λ i “ λ i r Σ s , so that Λ “diag p λ 1 , ¨ ¨ ¨ , λ d q and Σ “ PΛP T .
Revised: When A “ Σ “ Σ p X q , we simply write P “ P r Σ s , Λ “ Λ r Σ s , λ i “ λ i r Σ s , so that Λ “ diag p λ 1 , ¨ ¨ ¨ , λ d q and Σ “ PΛP T .

Original: Let P “ r P 1 P 2 ¨ ¨ ¨ P d s , where P i is the orthonormal basisvector corresponding to λ i .
Revised: Let P “ r P 1 P 2 ¨ ¨ ¨ P d s , where P i is the orthonormal basis vector corresponding to λ i .

Original: Rudimentary linear algebra yields λ k “ 1 n ř i
Revised: Rudimentary linear algebra yields λ k “ 1 n ř i

Original: p P
Revised: p P

Original: Tk X i q 2 for 1 ď k ď dand ||
Revised: Tk X i q 2 for 1 ď k ď d and ||

Original: X
Revised: X

Original: i || 

Revised: i ||

Original: “ ř k p P
Revised: “ ř k p P

Original: Tk X i q 2 for 1 ď i ď n .
Revised: Tk X i q 2 for 1 ď i ď n .

Original: Thus, it follows that
Revised: Thus, it follows that

Original: [Equation]
Revised: [Equation]

Original: That is, 0 ď tr r Λ s ď 1 is the average ℓ 2 norm (squared) of the X
Revised: That is, 0 ď tr r Λ s ď 1 is the average ℓ 2 norm (squared) of the X

Original: i ’s, and we simply write it as tr .
Revised: i ’s, and we simply write it as tr .

Original: Recall that it is assumed that all the X i ’s live in B d .
Revised: Recall that it is assumed that all the X i ’s live in B d .

Original: In practice, this is enforced by assuming anupper bound B on the norms and scaling down all X i by B .
Revised: In practice, this is enforced by assuming an upper bound B on the norms and scaling down all X i by B .

Original: As one often uses a conservativelylarge B , typical values of tr can be much smaller than 1 , so a trace-sensitive bound would bemore desirable.
Revised: As one often uses a conservatively large B , typical values of tr can be much smaller than 1 , so a trace-sensitive algorithm would be more desirable.

Original: Indeed, Amin et al.
Revised: Indeed, Amin et al.

Original: [2] take this approach, describing an algorithm with error˜ O p d 3 { 4 ? tr {? n ` ?
Revised: [2] take this approach, describing an algorithm with error 1 ˜ O p d 3 { 4 ? tr {? n ` ?

Original: d { n q under approximate-DP 2 .
Revised: d { n q under zCDP 2 .

Original: Note that the ?
Revised: Note that the ?

Original: d { n term inherits from meanestimation and the first term is the “extra” difficulty for covariance estimation.
Revised: d { n term inherits from mean estimation and the first term is the “extra” difficulty for covariance estimation.

Original: In this paper, weimprove this term to d 1 { 4 ? tr {?
Revised: In this paper, we improve this term to d 1 { 4 ? tr {?

Original: n (we have a similar, albeit lesser, improvement under pure-DP;see Appendix G).
Revised: n (we have a similar, albeit lesser, improvement under pure-DP; see Appendix I).

Original: In fact, our worst-case bound above follows from this result by simply takingtr “ 1 , which then absorbs the ?

Original: d { n term for d ă ˜ O p n 2 q . Our algorithm is very simple: We firstestimate Λ using the Gaussian mechanism (this is the same as in [2, 19]), then we estimate P bydoing an eigendecomposition of Σ masked with Gaussian noise.
Revised: Our algorithm is very simple: We first estimate Λ using the Gaussian mechanism (this is the same as in [2, 22]), then we estimate P by doing an eigendecomposition of Σ masked with Gaussian noise.

Original: Intuitively, we obtain a ?
Revised: Intuitively, we obtain a ?

Original: d -factorimprovement over the iterative methods of [2, 19], because we can obtain all eigenvectors fromone noisy Σ , while the iterative methods must allocate the privacy budget to all d eigenvectors.
Revised: d -factor improvement over the iterative methods of [2, 22], because we can obtain all eigenvectors from one noisy Σ , while the iterative methods must allocate the privacy budget to all d eigenvectors.

Original: Our algorithm is also more efficient, performing just two eigendecompositions and one matrixmultiplication, whereas the algorithm in [2, 19] needs O p d q such operations.
Revised: Our algorithm is also more efficient, performing just two eigendecompositions and one matrix multiplication, whereas the algorithm in [2, 22] needs O p d q such operations.

Original: 1.3 Tail-sensitive Bounds
Revised: Implication to worst-case bounds. Covariance matrix has also been studied in the traditional worst-case setting, i.e., the bound should only depend on d and n . Dwork et al. [17] show that the ℓ 2 -sensitivity of Σ , i.e., max X „ X 1 } Σ p X q ´ Σ p X 1 q} F where X „ X 1 denotes two neighboring datasets that differ by one column, is O p 1 { n q .

Revised: Thus, the standard Gaussian mechanism achieves an error of ˜ O p d { n q by adding an independent Gaussian noise of scale ˜ O p 1 { n q to each of the d 2 entries of Σ .

Revised: By taking tr “ 1 , our trace-sensitive bound degenerates into ˜ O p d 1 { 4 {? n q .

Revised: Note that the ?

Revised: d { n term is dominated by d 1 { 4 {?

Revised: n for d ă ˜ O p n 2 q , which is the parameter regime that allows non-trivial utility (i.e., the error is less than 1 ).

Revised: To better understand the situation, it is instructive to compare covariance estimation with mean estimation (where data are also drawn from the ℓ 2 unit ball and the error is measured in ℓ 2 norm), as the hardness of covariance estimation lies between d -dimensional mean estimation (only estimating the diagonal entries of Σ ) and d 2 -dimensional mean estimation (treating Σ as a d 2 -dimensional vector).

Revised: This observation implies a lower bound r Ω p?

Revised: d { n q following from the same lower bound for mean estimation [19] 3 , and an upper bound ˜

Revised: O p d { n q attained by the Gaussian mechanism.

Revised: For d ă O p?

Revised: n q , Kasiviswanathan et al.

Revised: [23] prove a higher lower bound 4 ˜Ω p d { n q , which means that the complexity of covariance estimation is same as d 2 -dimensional mean estimation in the lowdimensional regime, so one cannot hope to beat the Gaussian mechanism for small d .

Revised: However, in the high-dimensional regime, our result indicates that the covariance problem is strictly easier, due to the correlations of the d 2 entries of Σ .

Revised: Another interesting consequence is that our error bound has utility for d up to ˜

Revised: O p n 2 q (utility is lost when the error is ˜ O p 1 q , as returning a zero matrix can already achieve this error).

Revised: This is the highest d that allows for any utility, since even mean estimation requires d ă ˜ O p n 2 q to have utility under zCDP [19, 10].

Revised: We pictorially show the currently known

Revised: (worst-case) upper and lower bounds in Figure 1.

Revised: It remains an interesting open problem to close the gap for r Ω p?

Revised: n q ă d ă ˜ O p n 2 q .

Revised: Through private communication with Aleksandar Nikolov, it is observed that the projection mechanism [31, 15] can also be shown to have error ˜

Revised: O p d 1 { 4 {?

Revised: n q when applied to the covariance problem.

Revised: In Appendix E, we make this connection more explicit, while also giving an efficient implementation.

Revised: However, the projection mechanism is not trace-sensitive.

Revised: 1.2 A Tail-sensitive Algorithm

Original: A trace-sensitive bound only makes use of the average ℓ 2 norm, which cannot capture the fulldistribution.
Revised: A trace-sensitive bound only makes use of the average ℓ 2 norm, which cannot capture the full distribution.

Original: Next, we design an algorithm with an error bound that more closely depends on thedistribution of the norms.
Revised: Next, we design an algorithm with an error bound that more closely depends on the distribution of the norms.

Original: We characterize this distribution using the τ -tail ( I p¨q is the indicatorfunction):
Revised: We characterize this distribution using the τ -tail ( I p¨q is the indicator function):

Original: [Equation]
Revised: [Equation]

Original: Note that γ p X , τ q decreases as τ increases.
Revised: Note that γ p X , τ q decreases as τ increases.

Original: In particular, γ p X , 0 q “ tr , γ p X , 1 q “ 0 .
Revised: In particular, γ p X , 0 q “ tr , γ p X , 1 q “ 0 .

Original: A common technique to reduce noise, at the expense of some bias, is to clip all the X i ’s
Revised: A common technique to reduce noise, at the expense of some bias, is to clip all the X i ’s

Original: so that theyhave norms at most τ , for some threshold τ .
Revised: so that they have norms at most τ , for some threshold τ .

Original: This yields an error of Noise p X , τ q ` γ p X , τ q , where
Revised: This yields an error of Noise p X , τ q ` γ p X , τ q , where

Original: Noise p X , τ q denotes the error bound of the mechanism when all the X i ’s have norm bounded byτ , and γ p X , τ q is the (additional) bias caused by clipping.
Revised: Noise p X , τ q denotes the error bound of the mechanism when all the X i ’s have norm bounded by τ , and γ p X , τ q is the (additional) bias caused by clipping.

Original: Opting for the better of the Gaussianmechanism or our trace-sensitive mechanism, we have
Revised: Opting for the better of the Gaussian mechanism or our trace-sensitive mechanism, we have

Original: [Equation]
Revised: [Equation]

Original: The technical challenge is therefore choosing a good τ in a differentially private manner.
Revised: The technical challenge is therefore choosing a good τ in a differentially private manner.

Original: We designa DP mechanism to choose the optimal τ up to a polylogarithmic multiplicative factor and anexponentially small additive term.
Revised: We design a DP mechanism to choose the optimal τ up to a polylogarithmic multiplicative factor and an exponentially small additive term.

Original: It also adaptively selects the better of Gaussian mechanism or the trace-sensitive mechanism depending on the relationship between d, n , and a privatized tr .
Revised: It also adaptively selects the better of Gaussian mechanism or the trace-sensitive mechanism depending on the relationship between d, n , and a privatized tr .

Original: Moreprecisely, our adaptive mechanism achieves an error of
Revised: More precisely, our adaptive mechanism achieves an error of

Original: [Equation]
Revised: [Equation]

Original: Note that this tail-sensitive bound is always no worse (modulo the 2 ´ dn term) than Noise p X , 1 q(i.e., without clipping), and can be much better for certain norm distributions.
Revised: Note that this tail-sensitive bound is always no worse (modulo the 2 ´ dn term) than Noise p X , 1 q (i.e., without clipping), and can be much better for certain norm distributions.

Revised: In particular, the tail-sensitive bound would work very well on many real datasets with skewed distributions, e.g., most data vectors have small norms with a few having large norms.

Original: For example, supposed “ n 3 { 4 , and a constant number of data vectors have ℓ 2 norm 1 while the others have norm n ´ 1 { 4 ,then ?
Revised: For example, suppose d “ n 3 { 4 , and a constant number of data vectors have ℓ 2 norm 1 while the others have norm n ´ 1 { 4 . Then ?

Original: tr “ Θ p n ´ 1 { 4 q , so Noise p X , 1 q takes the trace-sensitive bound, which is ˜
Revised: tr “ Θ p n ´ 1 { 4 q , so Noise p X , 1 q takes the trace-sensitive bound, which is ˜

Original: O p n ´ 9 { 16 q .
Revised: O p n ´ 9 { 16 q .

Original: On theother hand, (3) is at most ˜
Revised: On the other hand, (3) is at most ˜

Original: O p n ´ 13 { 16 q by taking τ “ n ´ 1 { 4 .
Revised: O p n ´ 13 { 16 q by taking τ “ n ´ 1 { 4 .

Original: This example also shows that such atail-sensitive bound could work very well on many real datasets with skewed distributions.

Original:  Related Work
Revised:  Related Work

Original: Mean estimation and covariance estimation are perhaps the most fundamental problems in statisticsand machine learning, and how to obtain the best estimates while respecting individual’s privacyhas attracted a lot of attention in recent years.
Revised: Mean estimation and covariance estimation are perhaps the most fundamental problems in statistics and machine learning, and how to obtain the best estimates while respecting individual’s privacy has attracted a lot of attention in recent years.

Original: Mean estimation under differential privacy is nowrelatively well understood, with the optimal worst-case error being ˜ Θ p ?
Revised: Mean estimation under differential privacy is now relatively well understood, with the optimal worst-case error being ˜Θ p?

Original: d { n q [17], achieved by thestandard Gaussian mechanism [15].
Revised: d { n q [19], achieved by the standard Gaussian mechanism [17].

Original: In contrast, thecomplexity of covariance estimation remainselusive.
Revised: In contrast, the covariance problem is more elusive.

Original: The trivial upper bound is ˜

Original: O p d { n q while there is no lower bound higher than ˜Ω p?

Original: d { n q . Wehave improved the upper bound to ˜ O p d 1 { 4 {? n q for the regime d ą r Ω p n 2 { 3 q . It would be interestingto see if it is possible to also improve the ˜
Revised: As indicated in Figure 1, its complexity is probably a piecewise linear (in the log-log scale) function.

Original: O p d { n q upper bound in lower dimensions.

Original: When most data have norms much smaller than the upper bound given a priori , the worst-casebounds above are no longer optimal.
Revised: When most data have norms much smaller than the upper bound given a priori , the worst-case bounds above are no longer optimal.

Original: In these cases, it is more desirable to have an error bound that isinstance-specific.
Revised: In these cases, it is more desirable to have an error bound that is instance-specific.

Original: Clipping is a common technique for mean estimation [3, 16, 4, 27, 24] and it isknown that running the Gaussian mechanism after clipping X with a certain quantile of the norms ofthe X
Revised: Clipping is a common technique for mean estimation [3, 18, 4, 34, 29] and it is known that running the Gaussian mechanism after clipping X with a certain quantile of the norms of the X

Original: i ’s achieves
Revised: i ’s achieves

Original: instance-optimality in a certain sense [3, 16].
Revised: instance-optimality in a certain sense [3, 18].

Original: However, for covariance estimation,we show in Appendix D that no quantile can be the optimal clipping threshold achieving the bound in(3).
Revised: However, for covariance estimation, we show in Appendix F that no quantile can be the optimal clipping threshold achieving the bound in

Original: Nevertheless, the bound in (3) is only achieving the optimal clipping threshold; we cannot saythat is instance-optimal, since Noise p¨q is not even known to be worst-case optimal.
Revised: Nevertheless, the bound in (3) is only achieving the optimal clipping threshold; we cannot say that is instance-optimal, since Noise p¨q is not even known to be worst-case optimal.

Original: Closely related to covariance estimation are the PCA problem and low-rank approximation.
Revised: Closely related to covariance estimation are the PCA problem and low-rank approximation.

Original: Insteadof finding all eigenvalues and eigenvectors, they only aim at finding the largest one or a few.
Revised: Instead of finding all eigenvalues and eigenvectors, they only aim at finding the largest one or a few.

Original: For theseproblems, iterative methods [2, 19, 30, 15, 10] should perform better than the Gaussian mechanismor our algorithm, both of which try to recover the full covariance matrix.
Revised: For these problems, iterative methods [2, 22, 38, 17, 11, 36] should perform better than the Gaussian mechanism or our algorithm, both of which try to recover the full covariance matrix.

Original: Many covariance estimation algorithms have been proposed under the statistical setting, where the X i ’s are i.i.d.
Revised: Many covariance estimation algorithms have been proposed under the statistical setting, where the X i ’s are i.i.d.

Original: samples drawn from a certain distribution, e.g., a multivariate Gaussian [17, 8, 7, 1,18, 23].
Revised: samples drawn from a certain distribution, e.g., a multivariate Gaussian [19, 9, 8, 1, 21, 28, 5, 25].

Original: Instead of the Frobenius error, many of them adopt the Mahalanobis error } r Σ ´ Σ } Σ : “} Σ ´ 1 { 2 r ΣΣ ´ 1 { 2 ´ I } F , which can be considered as a normalized version of the former.
Revised: Instead of the Frobenius error, many of them adopt the Mahalanobis error } r Σ ´ Σ } Σ : “ } Σ ´ 1 { 2 r ΣΣ ´ 1 { 2 ´ I } F , which can be considered as a normalized version of the former.

Original: It is knownthat λ d } A ´ Σ } Σ ď } A ´ Σ } F ď λ 1 } A ´ Σ } Σ , so when Σ D is well-conditioned, i.e., λ 1 { λ d “ O p 1 q ,any Frobenius error directly translates to a Mahalanobis error.
Revised: It is known that λ d } A ´ Σ } Σ ď } A ´ Σ } F ď λ 1 } A ´ Σ } Σ , so when Σ D is well-conditioned, i.e., λ 1 { λ d “ O p 1 q , any Frobenius error directly translates to a Mahalanobis error.

Original: However, for the Mahalanobis error,the more challenging question is how to deal with an ill-conditioned Σ , for which [17, 7] haveprovided elegant solutions for the case where D is a multivariate Gaussian.
Revised: However, for the Mahalanobis error, the more challenging question is how to deal with an ill-conditioned Σ , for which [19, 8] have provided elegant solutions for the case where D is a multivariate Gaussian.

Original: It would be interestingto see if their methods can be combined with the tail-sensitive techniques in this paper to solve thisproblem for other distribution families, in particular, heavy-tailed distributions.
Revised: It would be interesting to see if their methods can be combined with the tail-sensitive techniques in this paper to solve this problem for other distribution families, in particular, heavy-tailed distributions.

Revised: For the lower bound, very recently, Kamath et al.

Revised: [20] proved a similar lower bound for the low-dimensional regime as in [23] but under the statistical setting.

Original:  Preliminaries 
Revised:  Preliminaries 3.1 Differential Privacy

Original: 3.1 Differential Privacy

Original: We say that X , X 1 P R d ˆ n are neighbors if they differ by one column, denoted X „ X 1 .
Revised: We say that X , X 1 P R d ˆ n are neighbors if they differ by one column, denoted X „ X 1 .

Original: Definition 1 (Differential Privacy (DP) [14]) .
Revised: Definition 1 (Differential Privacy (DP) [16]) .

Original: For ε ą 0 and δ ě 0 , a randomized mechanism
Revised: For ε ą 0 and δ ě 0 , a randomized mechanism

Original: M : R d ˆ n Ñ Y satisfies p ε, δ q -DP if for any X „ X 1 and any S Ď Y , Pr r M p
Revised: M : R d ˆ n Ñ Y satisfies p ε, δ q -DP if for any X „ X 1 and any S Ď Y , Pr r M p

Original: X q P S s ďe ε ¨ Pr r M p X 1 q P S s ` δ .
Revised: X q P S s ď e ε ¨ Pr r M p X 1 q P S s ` δ .

Original: In particular, we call it pure-DP if δ “ 0 ; otherwise approximate-DP .
Revised: In particular, we call it pure-DP if δ “ 0 ; otherwise approximate-DP .

Original: Definition 2 (Concentrated Differential Privacy (zCDP) [9]) .
Revised: Definition 2 (Concentrated Differential Privacy (zCDP) [10]) .

Original: For ρ ą 0 , a randomized mechanism
Revised: For ρ ą 0 , a randomized mechanism

Original: M : R d ˆ n Ñ Y satisfies ρ
Revised: M : R d ˆ n Ñ Y satisfies ρ

Original: -zCDP if for any X „ X 1 , D α p M p
Revised: -zCDP if for any X „ X 1 , D α p M p

Original: X
Revised: X

Original: q|| M p X 1 qq ď
Revised: q|| M p X 1 qq ď

Original: ρ ¨ α for all α ą 1 ,where D α p M p
Revised: ρ ¨ α for all α ą 1 , where D α p M p

Original: X
Revised: X

Original: q|| M p X 1 qq is the α -Rényi divergence between M p X q and M p X 1 q .
Revised: q|| M p X 1 qq is the α -Rényi divergence between M p X q and M p X 1 q .

Original: The relationship between these DP definitions is as follows.
Revised: The relationship between these DP definitions is as follows.

Original: Pure-DP, also written as ε -DP, impliesε 2 2 -zCDP, which further implies ´ ε 2 2 ` ε b 2 log 1 δ , δ ¯ -DP for any δ ą 0 .
Revised: Pure-DP, also written as ε -DP, implies ε 2 2 -zCDP, which further implies [Equation]

Original: To preserve ε -DP for a query Q , a standard mechanism is to add independent Laplace noises withscale proportional to the (global) ℓ 1 -sensitivity of Q to each dimension.
Revised: To preserve ε -DP for a query Q , a standard mechanism is to add independent Laplace noises with scale proportional to the (global) ℓ 1 -sensitivity of Q to each dimension.

Original: Lemma 1 (Laplace Mechanism [12]) .
Revised: Lemma 1 (Laplace Mechanism [13]) .

Original: Given Q : R d ˆ n Ñ R k , let GS Q : “ max X „ X 1 } Q p X q ´
Revised: Given Q : R d ˆ n Ñ R k , let GS Q : “ max X „ X 1 } Q p X q ´

Original: Q p X 1 q} 1 .
Revised: Q p X 1 q} 1 .

Original: The mechanism M p X q “ Q p X q ` GS Q ε ¨ Y where Y „ Lap p 1 q k , preserves ε -DP.
Revised: The mechanism M p X q “ Q p X q ` GS Q ε ¨ Y where Y „ Lap p 1 q k , preserves ε -DP.

Original: The following composition property of ε -DP allows us to design algorithms in a modular fashion.
Revised: The following composition property of ε -DP allows us to design algorithms in a modular fashion.

Original: Lemma 2 (Basic Composition) .
Revised: Lemma 2 (Basic Composition) .

Original: If M is an adaptive composition of mechanisms M 1 , M 2 , . .
Revised: If M is an adaptive composition of mechanisms M 1 , M 2 , . .

Original: , M t ,where each M i satisfies ε i -DP, then M satisfies p ř
Revised: , M t , where each M i satisfies ε i -DP, then M satisfies p ř

Original: i ε i q -DP.
Revised: i ε i q -DP.

Original: For ρ -zCDP, the standard method is the Gaussian mechanism :
Revised: For ρ -zCDP, the standard method is the Gaussian mechanism :

Original: Lemma 3 (Gaussian Mechanism [9]) .
Revised: Lemma 3 (Gaussian Mechanism [10]) .

Original: Given Q : R d ˆ n Ñ R k , let GS Q : “ max X „ X 1 } Q p X q ´
Revised: Given Q : R d ˆ n Ñ R k , let GS Q : “ max X „ X 1 } Q p X q ´

Original: Q p X 1 q} 2 .
Revised: Q p X 1 q} 2 .

Original: The mechanism M p X q “ Q p X
Revised: The mechanism M p X q “ Q p X

Original: q
Revised: q

Original: ` GS Q ? 

Revised: ` GS Q ?

Original: ρ ¨ Y where Y „ N p 0 ,
Revised: ρ ¨ Y where Y „ N p 0 ,

Original: I k ˆ k q , preserves ρ -zCDP.
Revised: I k ˆ k q , preserves ρ -zCDP.

Original: It has been shown that the covariance matrix has an ℓ 2 -sensitivity of ? 2 n [7].
Revised: It has been shown that the covariance matrix has an ℓ 2 -sensitivity of ? 2 n [8].

Original: Thus, the Gaussianmechanism for covariance, denoted GaussCov , simply adds an independent Gaussian noise withscale 1 ?
Revised: Thus, the Gaussian mechanism for covariance, denoted GaussCov , simply adds an independent Gaussian noise with scale 1 ?

Original: ρn to each entry of Σ .
Revised: ρn to each entry of Σ .

Original: Considering that Σ is symmetric, symmetric noises also suffice, whichpreserve the symmetry of the privatized Σ .
Revised: Considering that Σ is symmetric, symmetric noises also suffice, which preserve the symmetry of the privatized Σ .

Original: More precisely, we draw a random noise matrix Wwhere w j,k „ N p 0 , 1 q i.i.d.
Revised: More precisely, we draw a random noise matrix W where w j,k „ N p 0 , 1 q i.i.d.

Original: for 1 ď j ď k ď d and w k,j “ w j,k , denoted as W „ SGW p d q .
Revised: for 1 ď j ď k ď d and w k,j “ w j,k , denoted as W „ SGW p d q .

Original: Then
Revised: Then

Original: GaussCov outputs r Σ Gau “ Σ ` 1 ?
Revised: GaussCov outputs r Σ Gau “ Σ ` 1 ?

Original: ρn ¨ W .
Revised: ρn ¨ W .

Original: A similar composition property exists for ρ -zCDP.
Revised: A similar composition property exists for ρ -zCDP.

Original: Lemma 4 (Composition Theorem [9]) .
Revised: Lemma 4 (Composition Theorem [10]) .

Original: If M is an adaptive composition of algorithms
Revised: If M is an adaptive composition of algorithms

Original: M 1 , M 2 , . .
Revised: M 1 , M 2 , . .

Original: , M t , where each M i satisfies ρ i
Revised: , M t , where each M i satisfies ρ i

Original: -zCDP, then M satisfies p ř
Revised: -zCDP, then M satisfies p ř

Original: i ρ i q
Revised: i ρ i q

Original: -zCDP.
Revised: -zCDP.

Original: 3.2 The Sparse Vector Technique
Revised: 3.2 The Sparse Vector Technique

Original: The Sparse Vector Technique ( SVT ) [13] has as input a sequence of scalar queries,f 1 p X q , f 2 p X q , . . .
Revised: The Sparse Vector Technique ( SVT ) [14] has as input a sequence of scalar queries, f 1 p X q , f 2 p X q , . . .

Original: , f t p X q , where each has sensitivity 1 , and a threshold T .
Revised: , f t p X q , where each has sensitivity 1 , and a threshold T .

Original: It aims to find thefirst query (if there is) whose answer is approximately above T .
Revised: It aims to find the first query (if there is) whose answer is approximately above T .

Original: See Appendix A for the detailedalgorithm.
Revised: See Appendix A for the detailed algorithm.

Original: The SVT has been shown to satisfy ε -DP with following utility guarantee.
Revised: The SVT has been shown to satisfy ε -DP with following utility guarantee.

Original: Lemma 5 (Extension of Theorem 3.24 in [14]) .
Revised: Lemma 5 (Extension of Theorem 3.24 in [16]) .

Original: With probability at least 1 ´ β , SVT returns a k suchthat, for any i ă k , f i p X q ď T ` 6 ε log p 2 t { β q , and if k ‰ t ` 1 , then f k p X q ě T ´ 6 ε log p 2 t { β q .
Revised: With probability at least 1 ´ β , SVT returns a k such that, for any i ă k , f i p X q ď T ` 6 ε log p 2 t { β q , and if k ‰ t ` 1 , then f k p X q ě T ´ 6 ε log p 2 t { β q .

Original: 3.3 ConcentrationInequalities
Revised: Concentration

Revised: Inequalities

Original: Lemma 6 ([21]) .
Revised: Lemma 6 ([26]) .

Original: Given Y „ N p 0 , I d ˆ d q , with probability at least 1 ´ β ,} Y } 2 ď η p d, β q : “ b d `
Revised: Given Y „ N p 0 , I d ˆ d q , with probability at least 1 ´ β ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 7 ([7, 21]) .
Revised: Lemma 7 ([8, 26]) .

Original: Given W „ SGW p d q , with probability at least 1 ´ β , 
Revised: Given W „ SGW p d q , with probability at least 1 ´ β , }

Original: [Equation]
Revised: [Equation]

Original: Also, with probability at least 1 ´ β ,
Revised: Also, with probability at least 1 ´ β ,

Original: [Equation]
Revised: [Equation]

Original:  Ignoring polylogarithmic factors, η p d, β q and υ p d, β q are both in ˜ O p ?
Revised: qq ` 6 log p 2 { β q . Ignoring polylogarithmic factors, η p d, β q and υ p d, β q are both in ˜ O p?

Original: d q , while ω p d, β q is in ˜ O p d q .
Revised: d q , while ω p d, β q is in ˜ O p d q .

Original: These concentration inequalities are very useful for error analysis.
Revised: These concentration inequalities are very useful for error analysis.

Original: For example, the bound on } W } Fimmediately implies that GaussCov has error 1 ?
Revised: For example, the bound on } W } F immediately implies that GaussCov has error 1 ?

Original: ρn ¨ ω p d, β q “ ˜ O p d { n q .
Revised: ρn ¨ ω p d, β q “ ˜ O p d { n q .

Original:  Trace-sensitive Algorithm
Revised:  Trace-sensitive Algorithm

Original: The state-of-the-art trace-sensitive algorithm [2] first obtains an estimate of the eigenvalues, and theniteratively finds the eigenvectors by the exponential mechanism (EM), so we denote this algorithmas EMCov .
Revised: The state-of-the-art trace-sensitive algorithm [2] first obtains an estimate of the eigenvalues, and then iteratively finds the eigenvectors by the exponential mechanism (EM), so we denote this algorithm as EMCov .

Original: Under approximate-DP, it has an error of ˜ O p d 3 { 4 ? tr {? n ` ?
Revised: Under zCDP, it has an error of ˜ O p d 3 { 4 ? tr {? n ` ?

Original: d { n q .
Revised: d { n q .

Original: Below, we presentan algorithm that is simpler, faster, and more accurate, improving the trace-dependent term by a161 ?
Revised: Below, we present an algorithm that is simpler, faster, and more accurate, improving the trace-dependent term by a ?

Original: d -factor.
Revised: d -factor.

Original: The first step of our algorithm SeparateCov (shown in Algorithm 1) is basically the same as
Revised: The first step of our algorithm SeparateCov (shown in Algorithm 1) is basically the same as

Original: EMCov , where we obtain an estimate of the eigenvalues with half of the privacy budget.
Revised: EMCov , where we obtain an estimate of the eigenvalues with half of the privacy budget.

Original: [2] usesthe Laplace mechanism for pure-DP; for zCDP, we use the Gaussian mechanism, which relies onthe ℓ 2 -sensitivity of Λ , which we provide in Lemma 10 in the Appendix B. For the eigenvectors, weuse GaussCov to obtain a privatized r Σ Gau with the other half of the privacy budget, and perform aneigendecomposition.
Revised: [2] uses the Laplace mechanism for pure-DP; for zCDP, we use the Gaussian mechanism, which relies on the ℓ 2 -sensitivity of Λ , which we provide in Lemma 10 in the Appendix B. For the eigenvectors, we use GaussCov to obtain a privatized r Σ Gau with the other half of the privacy budget, and perform an eigendecomposition.

Original: Finally, we assemble the eigenvalues of eigenvectors to obtain a privatized Σ .
Revised: Finally, we assemble the eigenvalues of eigenvectors to obtain a privatized Σ .

Original: Itshould be clear that, after computing Σ , SeparateCov just needs two eigendecompositions and onefull matrix multiplication, plus some O p d 2 q -time operations.
Revised: It should be clear that, after computing Σ , SeparateCov just needs two eigendecompositions and one full matrix multiplication, plus some O p d 2 q -time operations.

Original: On the other hand, EMCov performs
Revised: On the other hand, EMCov performs

Original: O p d q eigendecompositions and matrix multiplications, plus a nontrivial sampling procedure for the
Revised: O p d q eigendecompositions and matrix multiplications, plus a nontrivial sampling procedure for the

Original: EM.
Revised: EM.

Original: Input: data X P B nd ; privacy parameter ρ ą 0 .

Original: 1: Λ Ð the eigenvalues of Σ “ 1 n XX T 2: r Λ Sep Ð Λ ` ? 2 ?

Original: ρn ¨ Y , where Y „ N p 0 , I d ˆ d q 3: r Σ Gau Ð GaussCov p X , ρ 2 q 4: r P Sep Ð P ”r Σ

Original: Gau ı 5: r Σ Sep Ð r P Sep r Λ

Original: Sep r P T

Original: Sep 6: return r Σ Sep

Original: That SeparateCov satisfies ρ -zCDP easily follows from the privacy of the Gaussian mechanism andthe composition property.
Revised: That SeparateCov satisfies ρ -zCDP easily follows from the privacy of the Gaussian mechanism and the composition property.

Original: The utility is given by the following theorem:
Revised: The utility is given by the following theorem:

Original: Theorem 1.
Revised: Theorem 1.

Original: Given any ρ ą 0 , for any X P B nd , and any β ą 0 , with probability at least 1 ´ β ,
Revised: Given any ρ ą 0 , for any X P B nd , and any β ą 0 , with probability at least 1 ´ β ,

Original: SeparateCov returns a r Σ Sep such that } r Σ Sep ´ Σ } F ď 2 1 . 25 ?
Revised: SeparateCov returns a r Σ Sep such that } r Σ Sep ´ Σ } F ď 2 1 . 25 ?

Original: tr ρ 1 { 4 ? n ¨ c υ ´ d, β¯ ` ? ρn ¨ η ´ d, β¯ “˜ O ´ d 1 { 4 ? tr ? n ` ?
Revised: [Equation] [Equation]

Original: dn ¯ .
Revised: n ¨

Original:  Remark While SeparateCov strictly improves over EMCov , it does not dominate GaussCov :
Revised: [Equation] Remark While SeparateCov strictly improves over EMCov , it does not dominate GaussCov :

Original: When tr ă ˜ O p d 3 { 2 { n q , SeparateCov is better; otherwise, GaussCov is better.
Revised: When tr ă ˜ O p d 3 { 2 { n q , SeparateCov is better; otherwise, GaussCov is better.

Original: EMCov is betterthan GaussCov for a smaller trace range: tr ă ˜ O p?
Revised: EMCov is better than GaussCov for a smaller trace range: tr ă ˜ O p?

Original: d { n q .
Revised: d { n q .

Revised: Theorem 1 implies our worst-case bound by taking tr “ 1 :

Revised: Theorem 2.

Revised: Given any ρ ą 0 , for any X P B n d , and any β ą 0 , with probability at least 1 ´ β ,

Revised: SeparateCov returns a r Σ Sep such that } r Σ Sep ´ Σ } F “ ˜ O ´ d 1 { 4 ? n ` ?

Revised: dn ¯ .

Original:  Tail-sensitive Algorithm 
Revised:  Tail-sensitive Algorithm 5.1 Clipped Covariance

Original: 5.1 Clipped Covariance

Original: Clipping is a common technique to reduce the sensitivity of functions at the expense of some bias.
Revised: Clipping is a common technique to reduce the sensitivity of functions at the expense of some bias.

Original: Given τ ě 0 and a vector X P R d , let Clip p X, τ q “ min ´ 1 , τ } X }¯ ¨ X .
Revised: Given τ ě 0 and a vector X P R d , let Clip p X, τ q “ min ´ 1 , τ } X }¯ ¨ X .

Original: Similarly, for any X P R d ˆ n ,
Revised: Similarly, for any X P R d ˆ n ,

Original: Clip p X , τ q denotes the matrix whose columns have been clipped to have norm at most τ .
Revised: Clip p X , τ q denotes the matrix whose columns have been clipped to have norm at most τ .

Original: Clippingcan be applied to both GaussCov and SeparateCov with a given τ : just run the mechanism on1 τ ¨ Clip p X , τ q and scale the result back by τ 2 .
Revised: Clipping can be applied to both GaussCov and SeparateCov with a given τ : just run the mechanism onτ ¨ Clip p X , τ q and scale the result back by τ 2 .

Original: We denote the clipped versions of the two mechanismsas ClipGaussCov and ClipSeparateCov , respectively.
Revised: We denote the clipped versions of the two mechanisms as ClipGaussCov and ClipSeparateCov , respectively.

Original: The following lemma bounds the bias caused by clipping in terms of the τ -tail as defined in (1).
Revised: The following lemma bounds the bias caused by clipping in terms of the τ -tail as defined in (1).

Original: Lemma 8. 
Revised: Lemma 8. } Σ

Original: [Equation]
Revised: [Equation]

Original: X , q .
Revised: ´

Original:  Thus, running the better of ClipGaussCov and ClipSeparateCov yields a total error of
Revised: [Equation] [Equation] Thus, running the better of ClipGaussCov and ClipSeparateCov yields a total error of

Original: Noise p X , τ, ρ, β q ` γ p X , τ q , where [Equation] ˆ
Revised: Noise p X , τ, ρ, β q ` γ p X , τ q , where Noise p X , τ, ρ, β q

Original: [Equation]
Revised: [Equation]

Original: ˙¸
Revised: min

Original: [Equation]
Revised: [Equation]

Original: which is the exact version of (2).
Revised: which is the exact version of (2).

Original: Note that the trace-sensitive term is only scaled by τ , which followsfrom the proof of Theorem 1 when all X i live in τ ¨ B d .
Revised: Note that the trace-sensitive term is only scaled by τ , which follows from the proof of Theorem 1 when all X i live in τ ¨ B d .

Original: Ideally, we would like to find the optimal noise-bias trade-off, i.e., achieving an error ofmin τ p Noise p X , τ q ` γ p X , τ qq .
Revised: Ideally, we would like to find the optimal noise-bias trade-off, i.e., achieving an error of min τ p Noise p X , τ q ` γ p X , τ qq .

Original: Two issues need to be addressed towards this goal: The first, minor,issue is that tr is sensitive, so we cannot use it directly to decide whether to use ClipGaussCovor ClipSeparateCov .
Revised: Two issues need to be addressed towards this goal: The first, minor, issue is that tr is sensitive, so we cannot use it directly to decide whether to use ClipGaussCov or ClipSeparateCov .

Original: This can be addressed by using a privatized upper bound of tr .
Revised: This can be addressed by using a privatized upper bound of tr .

Original: The morechallenging problem is how to find the optimal τ in a DP fashion.
Revised: The more challenging problem is how to find the optimal τ in a DP fashion.

Original: This problem has been well studiedfor the clipped mean estimator [16, 3], where it can be shown that setting τ to be the ˜ O p?
Revised: This problem has been well studied for the clipped mean estimator [18, 3], where it can be shown that setting τ to be the ˜ O p?

Original: d q -thlargest } X i } 2 results in the optimal noise-bias trade-off [16].
Revised: d q -th largest } X i } 2 results in the optimal noise-bias trade-off [18].

Original: Then the problem boils down to findinga privatized quantile, for which multiple solutions exist [16, 11, 25, 5, 29].
Revised: Then the problem boils down to finding a privatized quantile, for which multiple solutions exist [18, 12, 32, 6, 37].

Original: For the clipped meanestimator, using such a quantile of the norms results in the optimal trade-off because Noise p X , τ qtakes the simple form ˜ O p τ ?
Revised: For the clipped mean estimator, using such a quantile of the norms results in the optimal trade-off because Noise p X , τ q takes the simple form ˜ O p τ ?

Original: d { n q .
Revised: d { n q .

Original: In fact, if we only had ClipGaussCov , setting τ to be the ˜
Revised: In fact, if we only had ClipGaussCov , setting τ to be the ˜

Original: O p
Revised: O p

Original: d q -thlargest } X i } 2 would also yield an optimal trade-off, as ClipGaussCov is really just clipped mean ind 2 dimensions.
Revised: d q -th largest } X i } 2 would also yield an optimal trade-off, as ClipGaussCov is really just clipped mean in d 2 dimensions.

Original: However, due to the trace-sensitive noise term, it is no longer the case.
Revised: However, due to the trace-sensitive noise term, it is no longer the case.

Original: In Appendix D,we give examples showing that no quantile, whose rank may arbitrarily depend on d, n, tr , can achievean optimal trade-off even ignoring polylogarithmic factors.
Revised: In Appendix F, we give examples showing that no quantile, whose rank may arbitrarily depend on d, n, tr , can achieve an optimal trade-off even ignoring polylogarithmic factors.

Original: It thus calls for a new threshold-findingmechanism, which we describe next.
Revised: It thus calls for a new threshold-finding mechanism, which we describe next.

Original: 5.2 Adaptive Covariance: Finding the Optimal Clipping Threshold
Revised: 5.2 Adaptive Covariance: Finding the Optimal Clipping Threshold

Original: Our basic idea is to try successively smaller values τ “ 1 , 12 , 14 , . . . .
Revised: Our basic idea is to try successively smaller values τ “ 1 , 12 , 14 , . . . .

Original: As we reduce τ , the noisedecreases while the bias increases.
Revised: As we reduce τ , the noise decreases while the bias increases.

Original: We should stop when they are approximately balanced, whichwould yield a near-optimal τ .
Revised: We should stop when they are approximately balanced, which would yield a near-optimal τ .

Original: To do so in a DP manner, we need to quantify the noise and bias.
Revised: To do so in a DP manner, we need to quantify the noise and bias.

Original: Consider the bias first.
Revised: Consider the bias first.

Original: Given a τ ,we divide the interval p τ, 1 s into sub-intervals p τ, 2 τ s , p 2 τ, 4 τ s , . . .
Revised: Given a τ , we divide the interval p τ, 1 s into sub-intervals p τ, 2 τ s , p 2 τ, 4 τ s , . . .

Original: , p 12 , 1 s .
Revised: , p 12 , 1 s .

Original: For any X P X such that} X } 2 P p 2 s , 2 s ` 1 s , let q X “ Clip p X, τ q
Revised: For any X P X such that } X } 2 P p 2 s , 2 s ` 1 s , let q X “ Clip p X, τ q

Original: and then by Lemma 8,
Revised: and then by Lemma 8,

Original: [Equation]
Revised: [Equation]

Original: That is, clipping X can at most lead to 1 n ¨ p 2 2 s ` 2 ´ τ 2 q bias.
Revised: That is, clipping X can at most lead to 1 n ¨ p 2 2 s ` 2 ´ τ 2 q bias.

Original: Besides, since } X } 2 P p 2 s , 2 s ` 1 s , wehave
Revised: Besides, since } X } 2 P p 2 s , 2 s ` 1 s , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, given X , for any s P Z , we define Count s p X q : “ ˇˇ␣ X i : } X i } 2 P p 2 s ,2 s
Revised: Then, given X , for any s P Z , we define Count s p X q : “ ˇˇ␣ X i : } X i } 2 P p [Equation] s ,

Original: [Equation]
Revised: [Equation]

Original: 1 s (ˇˇ .
Revised: s (ˇˇ .

Original: It is easy tosee for any X „ X 1 , Count s differs by at most 1 , so does the sum of any subset of Count s ’s. Wecan define an upper bound on the bias: y Bias p X , τ q : “ 1 n ¨ ř s ă 0 s “ log 2 p τ q Count s ¨ p 2 2 s ` 2 ´ τ 2 q .
Revised: It is easy to see for any X „ X 1 , Count s differs by at most 1 , so does the sum of any subset of Count s ’s. We can define an upper bound on the bias: y Bias p X , τ q : “ 1 n ¨ ř s ă 0 s “ log 2 p τ q Count s ¨ p 2 2 s ` 2 ´ τ 2 q .

Original: Letq X “ Clip p X , τ q .
Revised: Let q X “ Clip p X , τ q .

Original: By (5) and (6), we have
Revised: By (5) and (6), we have

Original: [Equation]
Revised: [Equation]

Original: By the property of Count s ’s, given any τ , the sensitivity of y Bias p¨ , τ q is bounded by 1 n .
Revised: By the property of Count s ’s, given any τ , the sensitivity of y Bias p¨ , τ q is bounded by 1 n .

Original: Now we turn to the noise.
Revised: Now we turn to the noise.

Original: Recall that Noise p X , τ, ρ, β q is the smaller of two parts.
Revised: Recall that Noise p X , τ, ρ, β q is the smaller of two parts.

Original: The first part
Revised: The first part

Original: GaussNoise p τ, ρ, β q : “ τ 2 ¨ 1 ?
Revised: GaussNoise p τ, ρ, β q : “ τ 2 ¨ 1 ?

Original: ρn ¨ ω p d, β q is independent of X , so can be used directly.
Revised: ρn ¨ ω p d, β q is independent of X , so can be used directly.

Original: The secondpart depends on tr , is thus sensitive.
Revised: The second part depends on tr , is thus sensitive.

Original: Since its sensitivity is 1 n , we can easily privatize it by adding a
Revised: Since its sensitivity is 1 n , we can easily privatize it by adding a

Original: Gaussian noise of scale Θ
Revised: Gaussian noise of scale Θ

Original: [Equation]
Revised: [Equation]

Original: ¯ .
Revised: ¯ .

Original: For technical reasons, we need to use an upper bound, so we addΘ ´ log p 1 { β q ?
Revised: For technical reasons, we need to use an upper bound, so we add Θ ´ log p 1 { β q ?

Original: ρn ¯ to it so as to obtain a privatized p tr ě tr .
Revised: ρn ¯ to it so as to obtain a privatized p tr ě tr .

Original: Then we set
Revised: Then we set

Original: SeparateNoise p p tr , τ, ρ, β q : “ τ ¨ 2 1 .

Original: [Equation]
Revised: [Equation]

Original: Input: data X P B nd ; privacy parameter ρ ą 0 ; high probablity parameter β .

Original: 1: ˜ r Ð PrivRadius p X , ?

Original: ρ 2 , β 8 , 2 ´ 2 dn q 2: r X Ð Clip p X , ˜ r q 3: r tr Ð 1 n ř i } r X i }4: p tr Ð min ´r tr

Original: [Table]

Original: r 2 ?

Original: ρn

Original: [Table]

Original: N p 0 ,

Original: [Table]

Original: q ` 2 ? 2˜ r 2 ?

Original: ρn ¨ a log p 8 { β q ,

Original: [Table]

Original: r¯ 5:

Original: [Table]

Original: log

Original: [Table]

Original: p

Original: [Table]

Original: q

Original: [Table]

Original: 1 ´ SVT ´!

Original: Diff ´ r X , p tr , ˜ r, ρ 2 , β¯ , Diff ´ p r X ,  p tr , ˜ r 2 , ρ 2 , β¯ , . . . , Diff ´ pr X , p tr , 2 ´ dn , ρ 2 , β¯) , 0 , ? ρ ? ¯ 6:
Revised: and use { Noise p p tr , τ, ρ, β q : “ min ` GaussNoise p τ, ρ, β q , SeparateNoise p p tr , τ, ρ, β q ˘ as a DP upper bound of Noise p X , τ, ρ, β q .

Original: [Table]

Original: Ð min ´

Original: [Table]

Original: 1 , ˜ r ¯ 7: if SeparateNoise p p tr , ˜ τ, ρ 2 , β 2 q ě GaussNoise p ˜ τ, ρ 2 , β 2 q 8: r Σ Ada Ð ClipGaussCov p r X , ρ 2 , ˜ τ

Original: q 9: else 10: r Σ Ada Ð ClipSeparateCov p

Original: r X , ρ 2 , ˜ τ

Original: q 11: return r Σ Ada and use { Noise p p tr , τ, ρ, β q : “ min ` GaussNoise p τ, ρ, β q , SeparateNoise p p tr , τ, ρ, β q ˘ as a DP upperbound of Noise p X , τ, ρ, β q .

Original: Note that given p tr , { Noise p p tr , τ, ρ, β q is independent of X .
Revised: Note that given p tr , { Noise p p tr , τ, ρ, β q is independent of X .

Original: Finally, we run SVT on the following sequence of sensitivity-1 queries with T “ 0 :
Revised: Finally, we run SVT on the following sequence of sensitivity-1 queries with T “ 0 :

Original: Diff p X , p tr , τ, ρ, β q : “ n ¨ ´ y Bias p X , τ q ´ { Noise p p tr , τ, ρ, β q ¯ , τ “

Original: [Equation]
Revised: [Equation]

Original: The SVT would return a τ that balances the bias and noise.
Revised: The SVT would return a τ that balances the bias and noise.

Original: After finding such a τ , we choose to run either GaussCov or SeparateCov by comparing GaussNoise p τ, ρ, β q and SeparateNoise p p tr , τ, ρ, β q .
Revised: After finding such a τ , we choose to run either GaussCov or SeparateCov by comparing GaussNoise p τ, ρ, β q and SeparateNoise p p tr , τ, ρ, β q .

Original: As the sequence consists of dn queries, SVT has an error of O p log p dn qq , which, as we will show,affects the optimality by a logarithmic factor.
Revised: As the sequence consists of dn queries, SVT has an error of O p log p dn qq , which, as we will show, affects the optimality by a logarithmic factor.

Original: Meanwhile, the smallest τ we search over will inducean additive 2 ´ dn error.
Revised: Meanwhile, the smallest τ we search over will induce an additive 2 ´ dn error.

Original: The algorithm above can almost give us the desired error bound in (3), except that one thing maygo wrong: The SVT introduces an error that is a logarithmic factor larger than the optimum, butat least r Ω p 1 { n q .
Revised: The algorithm above can almost give us the desired error bound in (3), except that one thing may go wrong: The SVT introduces an error that is a logarithmic factor larger than the optimum, but at least r Ω p 1 { n q .

Original: This would be fine as long as there is one X i with } X i } 2 ě r Ω p 1 q , so that theoptimum error is r Ω p 1 { n q .
Revised: This would be fine as long as there is one X i with } X i } 2 ě r Ω p 1 q , so that the optimum error is r Ω p 1 { n q .

Original: However, when all the X i ’s have very small norms, say 1 { n 2 , the r Ω p 1 { n qerror from SVT would not preserve optimality.
Revised: However, when all the X i ’s have very small norms, say 1 { n 2 , the r Ω p 1 { n q error from SVT would not preserve optimality.

Original: To address this issue, we first find the radiusrad p X q “ max i } X i } 2 , and use it to clip X .
Revised: To address this issue, we first find the radius rad p X q “ max i } X i } 2 , and use it to clip X .

Original: The following lemma shows that, under DP, it ispossible to find a 2-approximation of rad p X q plus an additive b so that only O p log log p 1 { b qq vectorsare clipped.
Revised: The following lemma shows that, under DP, it is possible to find a 2-approximation of rad p X q plus an additive b so that only O p log log p 1 { b qq vectors are clipped.

Original: This allows us to set b “ 2 ´ dn while only incurring an O p log dn q error.
Revised: This allows us to set b “ 2 ´ dn while only incurring an O p log dn q error.

Original: Nicely, theymatch the additive and multiplicative errors that already exist from the SVT, so there is no asymptoticdegradation in the optimality.
Revised: Nicely, they match the additive and multiplicative errors that already exist from the SVT, so there is no asymptotic degradation in the optimality.

Revised: Input: data X P B nd ; privacy parameter ρ ą 0 ; high probablity parameter β .

Revised: [Table]

Revised: Ð PrivRadius p

Original: Lemma 9 ([11]) .
Revised: [Table] dn q [Table] r

Revised: [Table]

Revised: Ð Clip

Revised: [Table]

Revised: r

Revised: [Table]

Revised: ´r

Revised: [Table]

Revised: ¯

Revised: [Table]

Revised: ¯

Revised: [Table]

Revised: p

Revised: [Table]

Revised: dn

Revised: [Table]

Revised: ¯)

Revised: [Table]

Revised: [Table]

Revised: ¯

Revised: [Table]

Revised: min

Revised: [Table]

Revised: 7: if SeparateNoise p p tr , ˜

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ě GaussNoise p

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 8: r Σ Ada Ð ClipGaussCov p

Revised: [Table]

Revised: X ,

Revised: [Table]

Revised: [Table]

Revised: τ q 9: else 10: r Σ Ada Ð ClipSeparateCov p

Original:  For any ε ą 0 , β ą 0 and b ą 0 , given X P B nd , with probability at least1 ´ β , PrivRadius returns a ˜ r “ PrivRadius p X , ε, β, b
Revised: r X , ρ 2 , ˜ τ q 11: return r Σ Ada Lemma 9 ([12]) . For any ε ą 0 , β ą 0 and b ą 0 , given X P B nd , with probability at least 1 ´ β , PrivRadius returns a ˜ r “ PrivRadius p X , ε, β, b

Original: q such that ˜ r ď 2 ¨ rad p X q ` b and 
Revised: q such that ˜ r ď 2 ¨ rad p X q ` b and |t} X [Equation] [Equation] | “ O

Original: [Equation]
Revised: [Equation]

Revised: ¯ .

Original: The complete algorithm is given in Algorithm 2.
Revised: The complete algorithm is given in Algorithm 2.

Original: Its privacy follows from the privacy of PrivRadius ,
Revised: Its privacy follows from the privacy of PrivRadius ,

Original: SVT, GaussCov , SeparateCov , and the composition theorem of zCDP; its utility is analyzed in thefollowing theorem:
Revised: SVT, GaussCov , SeparateCov , and the composition theorem of zCDP; its utility is analyzed in the following theorem:

Original: Theorem 2.
Revised: Theorem 3.

Original: Given any ρ ą 0 and β ą 0 , for any X P B nd , with probability at least 1 ´ β ,
Revised: Given any ρ ą 0 and β ą 0 , for any X P B nd , with probability at least 1 ´ β ,

Original: AdaptiveCov returns a r Σ Ada such that
Revised: AdaptiveCov returns a r Σ Ada such that

Original: [Equation]
Revised: [Equation]

Original:  Experiments
Revised:  Experiments

Original: We conducted experiments  to evaluate our algorithms on both synthetic and real-world datasets.
Revised: We conducted experiments 5 to evaluate our algorithms on both synthetic and real-world datasets.

Original: Wecompare SeparateCov and AdaptiveCov against GaussCov [15], EMCov [2].
Revised: We compare SeparateCov and AdaptiveCov against GaussCov [17], EMCov [2].

Original: We implemented
Revised: We implemented

Original: EMCov in Python following the pseudo-code provided in [2] and the descriptions of the samplingalgorithm in [20].
Revised: EMCov in Python following the pseudo-code provided in [2] and the descriptions of the sampling algorithm in [24].

Original: We also tested CoinPress [7], but since it is designed to minimize the Mahalanobiserror, it does not perform well when measured in Frobenius error; as such we omit it from the reportedresults.
Revised: We also tested CoinPress [8], but since it is designed to minimize the Mahalanobis error, it does not perform well when measured in Frobenius error. The two distance measures coincide when Σ is well-conditioned but in this case, CoinPress degenerates into GaussCov . Therefore, we omit it from the reported results.

Original: As a baseline, we include returning a zero matrix, which has error O p tr q , hence a trivialtrace-sensitive algorithm.
Revised: As a baseline, we include returning a zero matrix, which has error O p tr q , hence a trivial trace-sensitive algorithm.

Original: In all of the experiments below, the p ε, δ q -DP version of EMCov willbe used, where we fix δ “ 1 e ´ 10 and compose its steps for eigenvector estimation using advancedcomposition.

Original: ε is set according to the relationship described in Section 3.1 between ρ -zCDP andp ε, δ q -DP.

Original: When rad p X q is much smaller than 1 , it is unfair for GaussCov and EMCov , so we scaleall datasets such that 0 .
Revised: When rad p X q is much smaller than 1 , it is unfair for GaussCov and EMCov , so we scale all datasets such that 0 .

Original: 5 ď rad p X q ď 1 .
Revised: 5 ď rad p X q ď 1 .

Original: As a result, we do not need the step to obtain a privateradius in AdaptiveCov , either.
Revised: As a result, we do not need the step to obtain a private radius in AdaptiveCov , either.

Original: Each experiment is repeated 50 times, and we report the averageerror.
Revised: Each experiment is repeated 50 times, and we report the average error.

Original: Furthermore, we have also conducted experiments under pure-DP; the results can be found in Appendix H. [Figure] [Figure]
Revised: Furthermore, we have also conducted experiments under pure-DP; the results can be found in Appendix J.

Original: 6.1 Synthetic Datasets
Revised: 6.1 Synthetic Datasets

Original: We generate synthetic datasets by first following the method in [2], to obtain a matrix X “ ZU ,where U P R d ˆ d is sampled from U p 0 , 1 q , and Z P R n ˆ d is sampled from N p 0 , I q .
Revised: We generate synthetic datasets by first following the method in [2], to obtain a matrix X “ ZU , where U P R d ˆ d is sampled from U p 0 , 1 q , and Z P R n ˆ d is sampled from N p 0 , I q .

Original: Then the vectorsin X are adjusted to be centred at 0 and their norms scaled.
Revised: Then the vectors in X are adjusted to be centred at 0 and their norms scaled.

Original: In [2], the vectors are scaled to have unitℓ 2 norm; in our experiments, to better control tr and data skewness, we scale the norms so that theyfollow the Zipf’s law.
Revised: In [2], the vectors are scaled to have unit ℓ 2 norm; in our experiments, to better control tr and data skewness, we scale the norms so that they [Figure] [Figure] follow the Zipf’s law.

Original: More precisely, we divide the norms into N bins.
Revised: More precisely, we divide the norms into N bins.

Original: The number of vectors in thek -th bin is proportional to 1 { k s and their norm is 2 k ´ N .
Revised: The number of vectors in the k -th bin is proportional to 1 { k s and their norm is 2 k ´ N .

Original: The parameter s characterizes the skewness,which we fix as s “ 3 .
Revised: The parameter s characterizes the skewness, which we fix as s “ 3 .

Original: Note that N “ 1 corresponds to the unit-norm case with tr “ 1 .
Revised: Note that N “ 1 corresponds to the unit-norm case with tr “ 1 .

Original: The results on tr “ 1 case are shown in Figure 1, which correspond to the worst-case bounds.
Revised: The results on tr “ 1 case are shown in Figure 2, which correspond to the worst-case bounds.

Original: The ρhere is fixed at 0 .
Revised: The ρ here is fixed at 0 .

Original: and we examine the error growth w.r.t.
Revised: and we examine the error growth w.r.t.

Original: d for n “ 1000 , 4000 , 16000 .
Revised: d for n “ 1000 , 4000 , 16000 .

Original: The resultsgenerally agree with the theory: For low d , GaussCov is (slightly) better than SeparateCov , whilethe latter is much better for high d .
Revised: The results generally agree with the theory: For low d , GaussCov is (slightly) better than SeparateCov , while the latter is much better for high d .

Original: AdaptiveCov is able to choose the better of the two adaptively,with a small cost due to allocating some privacy budget to estimate tr .
Revised: AdaptiveCov is able to choose the better of the two adaptively, with a small cost due to allocating some privacy budget to estimate tr .

Original: Actually, if AdaptiveCov isgiven the precondition that all norms are 1 , this extra cost can be saved.
Revised: Actually, if AdaptiveCov is given the precondition that all norms are 1 , this extra cost can be saved.

Original: Next, we vary one of the parameters while fixing the others at their default values d “ 200 , n “ 50000 ,
Revised: Next, we vary one of the parameters while fixing the others at their default values d “ 200 , n “ 50000 ,

Original: N “ 4 and ρ “ 0 . 1 , and the results are reported in Figure 2.
Revised: N “ 4 and ρ “ 0 . 1 , and the results are reported in Figure 3.

Original: The most interesting case is Figure 2(a),where we increase N , hence reducing tr , which demonstrates the trace-sensitive bounds.
Revised: The most interesting case is Figure 3(a), where we increase N , hence reducing tr , which demonstrates the trace-sensitive bounds.

Original: Clearly,
Revised: Clearly,

Original: GaussCov is not trace-sensitive, while the other 4 methods are.
Revised: GaussCov is not trace-sensitive, while the other 4 methods are.

Original: In fact, returning the zero matrix is the best trace-sensitive algorithm if tr is sufficiently small.
Revised: In fact, returning the zero matrix is the best trace-sensitive algorithm if tr is sufficiently small.

Original: However, this may not be very meaningfulin practice, as N “ 2 5 means that most data have norm 2 ´ 31 but a few have norm 1 .
Revised: However, this may not be very meaningful in practice, as N “ 2 5 means that most data have norm 2 ´ 31 but a few have norm 1 .

Original: These fewmay be outliers and should be removed anyway.
Revised: These few may be outliers and should be removed anyway.

Original: Figure 2(b)–(d) shows that higher d , smaller n ,and smaller ρ all have similar effects, i.e., SeparateCov becomes better while GaussCov becomesworse, while AdaptiveCov is able to pick the better one.[Figure]
Revised: Figure 3(b)–(d) shows that higher d , smaller n , and smaller ρ all have similar effects, i.e., SeparateCov becomes better while GaussCov becomes worse, while AdaptiveCov is able to pick the better one.

Original: [Figure]
Revised: [Figure]

Original: 6.2 Real-world Datasets
Revised: 6.2 Real-world Datasets

Original: We also evaluate the algorithms on two real-world datasets.
Revised: We also evaluate the algorithms on two real-world datasets.

Original: The first dataset is the MNIST [22]dataset, which contains images of handwritten digits.
Revised: The first dataset is the MNIST [27] dataset, which contains images of handwritten digits.

Original: We use its training dataset which contains60 , 000 images represented as vectors in Z d 255 , where d “ 784 “ 28 ˆ 28 .
Revised: We use its training dataset which contains 60 , 000 images represented as vectors in Z d 255 , where d “ 784 “ 28 ˆ 28 .

Original: These vectors arenormalized by 255 ?
Revised: These vectors are normalized by 255 ?

Original: d in the experiments.
Revised: d in the experiments.

Original: We estimate r Σ using samples containing all the digits, wealso estimate r Σ corresponding to individual digits (reported in the Appendix H).
Revised: We estimate r Σ using samples containing all the digits, we also estimate r Σ corresponding to individual digits (reported in the Appendix J).

Original: In the first case, r Σcan be used for further dimensionality reduction analysis; in the second case, individual r Σ can beused for modelling the distributions of individual digits, which together can be used in a collectivemodel for classification (e.g. using a mixture model).
Revised: In the first case, r Σ can be used for further dimensionality reduction analysis; in the second case, individual r Σ can be used for modelling the distributions of individual digits, which together can be used in a collective [Figure] model for classification (e

Original:  The second dataset contains news commentarydata [26] consisting of approximately 15 , 000 articles, each containing 500 ´ 4300 words, which weconvert into vectors of various dimensions using the hashing trick implemented in the scikit-learnpackage.
Revised: [Figure] g. using a mixture model). The second dataset contains news commentary data [33] consisting of approximately 15 , 000 articles, each containing 500 ´ 4300 words, which we convert into vectors of various dimensions using the hashing trick implemented in the scikit-learn package.

Original: In this case, the estimated r Σ can be used to help with further feature selection for NLPmodels, for example.
Revised: In this case, the estimated r Σ can be used to help with further feature selection for NLP models, for example.

Original: These vectors are normalized to have unit ℓ 2 norm or normalized by the max ℓ 2norm.
Revised: These vectors are normalized to have unit ℓ 2 norm or normalized by the max ℓnorm.

Original: The experimental results on these two real dataset are shown in Figure 3 and 4, where we vary n, d ,and ρ , respectively.
Revised: The experimental results on these two real dataset are shown in Figure 4 and 5, where we vary n, d , and ρ , respectively.

Original: On these results, we see that GaussCov never outperforms SeparateCov , exceptfor a very small advantage in a few cases where we have tr “ 1 , a low d , and a high ρ .
Revised: On these results, we see that GaussCov never outperforms SeparateCov , except for a very small advantage in a few cases where we have tr “ 1 , a low d , and a high ρ .

Original: Anotherinteresting observation is that AdaptiveCov outperforms both GaussCov and SeparateCov in manycases, something that is not apparent on the synthetic datasets.
Revised: Another interesting observation is that AdaptiveCov outperforms both GaussCov and SeparateCov in many cases, something that is not apparent on the synthetic datasets.

Original: We believe that this is because thesereal datasets have heavier tails than the Zipf distribution (we used s “ 3 for Zipf), which makesthe adaptive clipping threshold selection more effective.
Revised: We believe that this is because these real datasets have heavier tails than the Zipf distribution (we used s “ 3 for Zipf), which makes the adaptive clipping threshold selection more effective.

Original: This really demonstrates the benefits of atail-sensitive bound. 
Revised: This really demonstrates the benefits of a tail-sensitive bound. Acknowledgements

Revised: This work has been supported by HKRGC under grants 16201819, 16205420, and 16205422.

Revised: We would like to thank Aleksandar Nikolov for helpful discussions on the projection mechanism and the anonymous reviewers who have made valuable suggestions on improving the presentation of the paper

Revised: Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] See Section 2.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes]

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] See URL in Section 6.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See URL in Section 6. 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] URLs are also provided where data were retrieved.

Revised: (b) Did you mention the license of the assets?

Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]