New Lower Bounds for Private Estimation and a Generalized Fingerprinting Lemma
Abstract
We prove new lower bounds for statistical estimation tasks under the constraint of ( ε, δ ) -differential privacy.
First, we provide tight lower bounds for private covariance estimation of Gaussian distributions.
We show that estimating the covariance matrix in Frobenius norm requires Ω( d 2 ) samples, and in spectral norm requires Ω( d 3 / 2 ) samples, both matching upper bounds up to logarithmic factors.
We prove these bounds via our main technical contribution, a broad generalization of the fingerprinting method [BUV14] to exponential families.
Additionally, using the private Assouad method of Acharya, Sun, and Zhang [ASZ21], we show a tight Ω( d/ ( α 2 ε )) lower bound for estimating the mean of a distribution with bounded covariance to α -error in ℓ 2 -distance.
Prior known lower bounds for all these problems were either polynomially weaker or held under the stricter condition of ( ε, 0) -differential privacy.
Introduction
The last several years have seen a surge of interest in algorithms for statistical estimation under the constraint of Differential Privacy (DP) [DMNS06].
We now have a rich algorithmic toolbox for private estimation of mean, covariance, and entire distributions, in a variety of settings.
While the community has enjoyed much success designing algorithms for estimation tasks, lower bounds have been much harder to come by, and consequently, the existing literature lacks a rigorous understanding of some core problems.
Lower bounds against the strong privacy constraint of pure DP (i.e., ( ε, 0) -DP) are generally not too challenging to prove, most often relying upon the well-known packing technique [HT10].
Such packing lower bounds are optimal in a broad range of settings (see, e.g., [BKSW19]), and thus are frequently effective at providing tight minimax sample complexity lower bounds.
However, pure DP is a very strong privacy constraint: in most practical circumstances, it suffices to only require the weaker privacy notion of approximate DP (i.e., ( ε, δ ) -DP)
[DKM + 06], in which case packing lower bounds are no longer applicable.
Proving lower bounds under approximate DP is much more challenging, and accordingly, the state of affairs is less satisfying.
There exist only a couple of techniques which apply in this setting, including the fingerprinting method [BUV14] and differentially private analogues of Le Cam’s method and Assouad’s lemma [ASZ18, ASZ21].
These techniques can be quite brittle and thus unable to prove lower bounds for some fairly basic settings.
For example, both techniques generally require that the underlying distribution has independent marginals and are thus ineffective at proving tight lower bounds for problems involving correlations such as private covariance estimation.
Furthermore, the fingerprinting method often needs the distribution’s marginals to have a precise functional form, such as Gaussian or Bernoulli.
These restrictions make the approaches somewhat brittle for proving lower bounds which differ too much from existing results.
Consequently, we lack approximate DP lower bounds for some fundamental settings which seem to qualitatively differ, including Gaussian covariance estimation and mean estimation of heavy-tailed distributions.
1.1 Our Results
We circumvent these barriers and fill a number of gaps in the literature by providing tight lower bounds for several estimation tasks under the constraint of approximate differential privacy.
Specifically, we address problems including covariance estimation for Gaussians and mean estimation with heavy-tailed data.
Our main technical tool for covariance estimation is a novel generalization of the fingerprinting method to exponential families, while we prove our mean estimation lower bound via DP Assouad’s lemma.
Our first result is a lower bound for covariance estimation in Frobenius norm.
Theorem 1.1 (Informal) .
For any α = O (1) , any ( ε, δ ) -DP mechanism with ε, δ ∈ [0 , 1] , and δ ≤
O (min (cid:8) 1 /n, d 2 / ( n log( n/d 2 )) (cid:9) ) that takes n samples from an arbitrary d -dimensional Gaussian N (0 , Σ) and outputs M ( X ) satisfying E X,M (cid:104) ∥
M ( X ) − Σ
∥
2 F (cid:105) ≤ α 2 requires n ≥ Ω (cid:16) d 2 αε (cid:17) .
We point out that the above lower bound also implies a lower bound for density estimation of
Gaussians with known mean and unknown covariance (see Theorem 1 .
1 of [DMR18]). This nearly matches the (cid:101) O (cid:0) d 2 (cid:1) upper bound of [KLSU19] up to polylogarithmic factors.
2 Previously, Ω (cid:0) d 2 (cid:1) lower bounds for Gaussian covariance estimation were only known under the more restrictive constraint of pure differential privacy [KLSU19, BKSW19].
Our result for Gaussian covariance estimation in Frobenius norm implies the following result for spectral estimation.
Theorem 1.2 (Informal) .
For any α = O (1 / √ d ) , any ( ε, δ ) -DP mechanism with ε, δ ∈ [0 , 1] , and δ ≤ O (min (cid:8) 1 /n, d 2 / ( n log( n/d 2 ))
(cid:9) ) that takes n samples from an arbitrary d -dimensional Gaussian N (0 , Σ) and outputs M ( X ) satisfying E X,M (cid:104)
∥
M ( X ) − Σ ∥ 22 (cid:105) ≤ α 2 requires n ≥ Ω (cid:16) d 1 .
5 αε (cid:17) .
In Theorem 3 . 8 of [KLSU19], the authors give guarantees for an individual iteration of a private recursive preconditioner.
In each iteration, the data is rescaled, resulting in constant-factor progress in privately reducing the condition number of the underlying covariance matrix to a constant.
Identifying the preconditioning matrix necessitates performing spectral estimation of the underlying covariance matrix.
Thus, the sample complexity upper bound of (cid:101) O
( d 3 / 2 ) which comes up in the guarantees of the preconditioning steps is also an upper bound for spectral estimation for α = O (1) .
This implies that our lower bound is tight up to polylogarithmic factors in the regime α = O (1 / √ d ) . Note that the sample complexity of the same problem sans privacy constraints is Θ ( d ) , and thus the cost of privacy is polynomial in the dimension d .
This is in contrast to Gaussian mean estimation
[Table]
[Table]
[Table] [Table] in ℓ 2 -norm and covariance estimation in Frobenius norm, which maintain their non-private sample complexities of O ( d ) and O ( d 2 ) , respectively. Ω( d 3 / 2 ) -sample lower bounds for approximate DP covariance estimation in spectral norm were previously known for worst-case distributions [DTTZ14].
Our work relaxes this assumption significantly to the Gaussian case while maintaining the same lower bound.
A priori, it was not clear that such a result was even true, as there are frequently gaps between the sample complexity of private estimation for worst-case versus well-behaved distributions for even basic settings, including distributions over the hypercube (see, e.g., Remark 6.4 of [BKSW19]).
This lower bound has implications for estimation tasks with scale-dependent error.
For example, consider Gaussian mean estimation in Mahalanobis distance.
A natural way to approach this problem is to first estimate the covariance matrix spectrally, and then estimate the mean after an appropriate rescaling (see, e.g., [KLSU19]).
Our lower bound shows that any such approach must incur the Ω( d 3 / 2 ) cost of spectral estimation, which is greater than the O ( d ) minimax sample complexity of the problem.
Indeed, some recent works [BGS + 21, LKO21] manage to circumvent this roadblock by adopting more direct (but computationally inefficient) methods.
Finally, we prove new lower bounds for mean estimation of distributions with bounded second moments.
Theorem 1.3 (Informal) .
For any α ≤ 1 , any ( ε, δ ) -DP mechanism with δ ≤ ε that takes n samples from an arbitrary distribution over R d with second moments bounded by 1 and outputs M ( X ) satisfying E X,M (cid:104)
[Equation]
X [Equation] 2 requires n ≥ Ω (cid:0) dα 2 ε (cid:1) .
Similar Θ( d/ ( α 2 ε )) upper and lower bounds were previously known for distributions with bounded second moments under the stricter constraint of pure differential privacy [BD14, KSU20, HKM21].
Also, a similar bound is shown in [KLZ21] for concentrated DP, but under a weaker moment assumption which involves only coordinate-wise projections.
Our result shows that the same bound holds under the weaker constraint of approximate DP for distributions satisfying the stronger moment assumption, and thus no savings can be obtained by relaxing the privacy notion.
In contrast to our other results, we show this lower bound via the DP Assouad method of [ASZ21], thus demonstrating the promise of different approaches for proving lower bounds for differentially private estimation.
1.2 Related Work
The fingerprinting technique for proving lower bounds was introduced by Bun, Ullman, and Vadhan [BUV14], which relied on the existence of fingerprinting codes [BS98, Tar08].
Since then, the technique has been significantly simplified and refined in a number of ways [SU17a, DSS + 15,
BSU17, SU17b, KLSU19, CWZ19], including the removal of fingerprinting codes and distilling the main technical component into the fingerprinting lemma .
Beyond the aforementioned settings of mean estimation of Gaussians and distributions over the hypercube [BUV14, SU15, DSS + 15, KLSU19], fingerprinting lower bounds have also been applied in settings including private empirical risk minimization [BST14] and private spectral estimation [DTTZ14].
Differentially private analogues of
Fano, Le Cam, and Assouad were first considered in the local DP setting [DJW13], and more recently under the constraint of central DP [ASZ18, ASZ21].
On the upper bounds side, the most relevant body of work is that which focuses on mean and covariance estimation for distributions satisfying a moment bound, see, e.g., [BD14, KV18, BKSW19,
BS19, KLSU19, CWZ19, KSU20, WXDX20, DFM + 20, BDKU20, AAK21, BGS + 21, KLZ21,
HLY21, KMS + 21, AL21].
Our lower bounds nearly match these upper bounds for Gaussian covariance estimation [KLSU19] and heavy-tailed mean estimation [KSU20, HKM21].
Previously, these bounds were conjectured to be optimal, albeit on somewhat shaky grounds including evidence from lower bounds proven under pure DP or non-rigorous connections with lower bounds in related settings.
Our results rigorously prove optimality and confirm these conjectures.
Other works study private statistical estimation in different and more general settings, including mixtures of
Gaussians [KSSU19, AAL21], graphical models [ZKKW20], discrete distributions [DHS15], and median estimation [AMB19, TVGZ20].
Some recent directions involve guaranteeing user-level privacy [LSY + 20, LSA + 21], or a combination of local and central DP for different users [ADK19].
See [KU20] for further coverage of DP statistical estimation.
1.3 Organization
We start by providing our generalized fingerprinting technique in Section 2.
Then we show its novel application for proving the lower bound for covariance estimation of Gaussians in Mahalanobis norm, followed by the lower bound for spectral norm estimation in Section 3.
Finally, we sketch the proof of the lower bound for mean-estimation of heavy-tailed distributions in Section 4.
Fingerprinting Proofs for Exponential Families
This section is split into three parts.
Section 2.1 provides an overview of existing fingerprinting proofs and motivation for our result, and introduces all the necessary notation.
Then, in Section 2.2, our new fingerprinting lemma is stated.
Finally, in Section 2.3, we give a general recipe of applying it.
2.1 Overview of Fingerprinting Proofs and Intuition for Our Result
At the heart of fingerprinting proofs lies the trade-off between accuracy and privacy.
For simplicity, assume we have a univariate data-generating distribution and coming from a family parameterized by its mean µ , which we want to estimate.
In contrast to minimax lower bounds in classical statistics (which are commonly obtained by constructing hard testing instances), fingerprinting proofs assume the existence of a subset of the domain of µ (commonly an interval [ ± R ] ) and that a value is drawn µ ∼ U ([ ± R ]) .
Then, n independent samples X 1 , . .
, X n are drawn from D µ .
Now, given an estimator f : R n → {± R } for µ , we consider the quantity Z i :=
c ( R, µ ) ( f ( X ) − µ ) ( X i − µ ) , ∀ i ∈ [ n ] , which measures the correlation between the output of f and the i -th sample ( c ( R, µ ) > 0 is a scaling factor that exists for purely technical reasons).
Defining Z := (cid:80) i ∈ [ n ] Z i , all proofs first show a lower bound L on E [ Z + ( f ( X ) − µ ) 2 ] , implying that our estimator cannot at the same time be accurate (with respect to the Mean-Squared-Error (MSE)) and exhibit low correlation with individual samples.
This component of the proof is traditionally called the fingerprinting lemma .
Privacy comes up in the second step of the proof, where we upper-bound the correlation terms E [ Z i ] .
We assume that f is a private mechanism with MSE at most L/ 2 .
Due to privacy, all correlation terms Z i should be upper-bounded by the same U =
U ( ε, δ ) .
Applying this for each sample leads to an inequality of the form nU ≥ L/ 2 ⇐⇒ n ≥ L/ (2 U ) , so the only thing that remains is to identify the appropriate parameter range for δ to prove the desired bound.
Our work is motivated by a fundamental weakness of existing fingerprinting proofs.
To prove lower bounds for high-dimensional distributions via fingerprinting, past works have assumed that the underlying distribution is a product distribution.
This made it possible to work component-wise using techniques for single-dimensional distributions.
This assumption is somewhat restrictive and makes it difficult to generalize the approach to settings with a richer correlation structure.
Our main technical contribution is providing a broad generalization of the celebrated fingerprinting method to exponential families (Definition A.5).
Distributions belonging in the same exponential family are parameterized by their natural parameter vector η , instead of their mean µ , for which it is uncommon to have a general closed-form expression.
Given n samples X 1 , . . .
, X n from p η , it suffices to know the values of the sufficient statistics T ( X 1 ) , . . .
, T ( X n ) instead of the samples themselves in order to estimate η (see, e.g., [Jor10]).
We denote the mean and covariance of T by µ T and Σ T , respectively.
Based on the above remarks, our fingerprinting lemma provides lower bounds for estimating η instead of µ .
We assume the existence of vectors η (1) , η (2) with η (1) j ≤ η (2) j , ∀ j ∈ [ k ] , and define the intervals I j := (cid:104) η (1) j , η (2) j (cid:105) for each coordinate j ∈ [ k ] .
Assuming that the hyperrectangle (cid:78) j ∈ [ k ] I j is a subset of H , we draw a natural parameter vector η uniformly at random from it.
Observe that I j may not be centered around the origin, which poses a minor technical difficulty.
To deal with this, we assume that our estimator does not estimate η itself, but rather the deviation of η from the midpoint m :=
( η (1) + η (2) ) / 2 .
The two problems are equivalent because any estimator [Equation] −
[Equation] − [Equation] ) / 2] for the latter problem can be transformed to an estimator for the former, simply by adding m to it (and vice-versa).
Defining R to be the width vector η (2) − η (1) , the correlation between the estimate for η j and the sample X
i becomes
[Equation]
Observe that now, Z ji measures the correlation between f and the sufficient statistics of the samples.
This is in contrast to older fingerprinting proofs, in which the Z ji measure correlation between f and the samples themselves.
We consider this to be natural in the context of exponential families.
Indeed, we know that, if we want to estimate the natural parameter vector of a distribution that belongs to an exponential family given a dataset X
=
( X 1 , . . .
, X n ) , it suffices to know the values
T ( X 1 ) , . . .
, T ( X n ) instead of the samples themselves (see [Jor10]).
Thus, despite f receiving X , correlation ends up being measured with respect to T ( X i ) , i ∈ [ n ] .
In addition to the above, we introduce three more correlation terms:
[Equation]
We motivate each of the above.
Z i describes the correlation between X i and our estimates for η .
Moreover, Z j describes the correlation between our estimate for η j and X .
Finally, Z describes the total correlation between the output of f and X .
Our fingerprinting lemma for exponential families will involve upper and lower-bounding the quantity
E η,X (cid:104) Z
+ ∥ f ( X ) − ( η − m )
∥ 22 (cid:105) .
2.2 The Main Lemma and Our Approach to Lower Bounds
We prove our new fingerprinting lemma in two parts.
First, we assume we have a distribution p η that belongs to an exponential family and we want to estimate η j whereas η − j is fixed (Lemma 2.1).
This setting is similar to the ones considered in [BSU17, KLSU19], which feature similar lemmas.
We then leverage Lemma 2.1 to get a result when the goal is to estimate all of η (Lemma 2.2).
Lemma 2.1.
Let p η be a distribution over S ⊆ R d belonging to an exponential family E ( T, h ) with natural parameter vector η ∈ H ⊆
R k .
We assume that I j is non-degenerate, and that η j is randomly generated by drawing η j ∼ U ( I j ) , whereas η − j is fixed.
Then, for any function f j : S n → [ ± R j / 2] that takes as input a dataset X :=
( X 1 , . . .
, X n ) ∼ p (cid:78) n η , we have:
[Equation]
Proof.
Let g j : (cid:78) j ∈ [ k ] I j
→ [ ± R j / 2] with g j ( η ) : = E X [ f j ( X )] .
g j can equivalently be written as:
[Equation]
R d . . . (cid:90) R d   (cid:89) i ∈ [ n ] p η ( x i )  
f j ( x 1 , . .
, x n ) dx 1 . . . dx n . Observe that:
[Equation]
[Equation]

[Equation] = ⇒ ∂ [Equation]
[Equation] [Equation] Calculating the partial derivative of g j with respect to η j and changing the order of differentiation and integration, we get:
[Equation]
where ( a ) relies on the observation that, conditioned on η j , η j − m j and (cid:80) i ∈ [ n ] ( T j ( X i ) − µ T
[Equation]
j ) are independent and E X (cid:104)(cid:80) i ∈ [ n ] ( T j ( X i ) − µ T,j ) (cid:105) =
[Equation]
We proceed to define the function G j : (cid:78) j ∈ [ k ] I j
→ R as:
[Equation]
Calculating the expectation of G j with respect to η j yields:
[Equation]
[Equation] where ( a ) uses integration by parts that m j has been defined to be equal to ( η (1) j + η (2) j ) / 2 .
Based on this, we remark that the inequality we wish to prove can be written in the form:
[Equation]
Appealing to linearity of expectation, we focus on the second term and get:
[Equation]
Substituting this to (1) yields the desired result.
■
We now generalize the previous lemma in the setting where all the components of η are drawn independently and uniformly at random from intervals.
Lemma 2.2.
Let p η be a distribution over S ⊆ R d belonging to an exponential family E ( T, h ) with natural parameter vector η ∈ H ⊆
R k .
We assume that η is randomly generated by drawing independently η j ∼ U ( I j ) , ∀ j ∈ [ k ] .
Then, for any function f : S n → (cid:78) j ∈ [ k ] [ ± R j / 2] that takes as input a dataset X :=
( X 1 , . . .
, X n ) ∼ p
[Equation]
n η , we have:
[Equation]
The proof is merely a component-wise application of Lemma 2.1.
2.3 From the Lemma to Lower Bounds
We leverage Lemma 2.2 to give a general recipe for proving lower bounds for private estimation.
The overall structure of our argument is similar to that of past fingerprinting proofs but, along the way, uses specific properties of exponential families to deal with the fact that we do not have a product distribution.
We assume the existence of an ( ε, δ ) -DP mechanism M : S n → (cid:78) j ∈ [ k ] [ ± R j / 2] with ε ∈ [0 , 1] and δ ≥ 0 such that, for any distribution p η with η ∈ (cid:78) j ∈
[ k ] I j , we have for X ∼ p (cid:78) n η : E X, [Equation]
For the above, Lemma 2.2 with f ≡ M yields:
[Equation]
Thus, our intent is to upper-bound the LHS by a function of n , which will allow us to identify a lower bound to get the desired accuracy.
To do so, we first need to upper-bound [Equation]
Lemma 2.3.
Conditioning on η , for any ε ∈ [0 , 1] , δ ≥ 0 , and T > 0 , it holds that:
[Equation]
where s ∈ R k with s j :=
[ R 2 j / 4 − ( η j − m j ) 2 ] [ M j ( X ∼ i ) − ( η j − m j )] , ∀ j ∈ [ k ] .
The proof involves splitting Z i in its positive and negative components and applying the definition of
DP to each of them.
The argument is standard and has appeared in previous works (e.g., see [SU17b]), but we need to apply specific properties of exponential families to deal with the fact that we don’t have a product distribution.
The proof is lengthy so we omit it.
Combining Lemma 2.3 with (3) yields the main result of this section:
Theorem 2.4.
Let p η be a distribution over S ⊆ R d belonging to an exponential family E ( T, h ) with natural parameter vector η ∈ H ⊆
R k .
Also, let η (1) , η (2) ∈ H and let I j := (cid:104) η
(1) j , η (2) j (cid:105) , ∀ j ∈ [ k ] be a collection of intervals and R :=
η (2) − η (1) , m :=
( η (1) + η (2) ) / 2 be the corresponding width and midpoint vectors, respectively.
Assume that (cid:78) j ∈ [ k ] I j ⊆ H and that η is drawn from the distribution U ( (cid:78) j ∈ [ k ] I j ) .
Moreover, assume that we have a dataset X ∼ p (cid:78) n η and an independently drawn point
X ′ i ∼ p
η
and X ∼ i denotes the dataset where X i has been replaced with X ′ i .
Finally, let M : S n → (cid:78) j ∈ [ k ] [ ± R j / 2] be an ( ε, δ ) -DP mechanism with ε ∈ [0 , 1] , δ ≥ 0 with
E X,M
[Equation]
Then, for any T > 0 , it holds that:
[Equation]
In order to apply the above theorem, the main idea is to identify values for T and δ such that
[Equation]
R
[Equation]
k ) (cid:105) dt , and 3 δTn ≤
[Equation]
R
[Equation]
The reasoning behind this is that the first and third terms of (2.4) both depend on T , with the latter term becoming smaller as T increases.
Thus, balancing these two terms involves identifying a value for T that is as small as possible and results in δT dominating the other term.
At the same time, we want the sum of those two terms to be smaller than the half of the RHS, so that we get a lower bound that is as tight as possible, leading to the constraint on δ .
The above result in the inequality
[Equation]
Consequently, obtaining the desired sample complexity lower bounds boils down to upper-bounding the term [Equation]
In the applications we will see, the upper bounds will be worst-case in terms of η , so the outer expectation E η [ · ] will be largely ignored.
Lower Bounds for Private Gaussian Covariance Estimation
Here, we provide lower bounds for covariance estimation of high-dimensional Gaussians with respect to the Mahalanobis and spectral norms under the constraint of ( ε, δ ) -DP.
3.1 Estimation with Respect to the Mahalanobis Norm
In this section, we characterize the sample complexity of private Gaussian covariance estimation with respect to the Mahalanobis norm under ( ε, δ ) -DP.
Previous lower bounds either assumed a stricter notion of privacy (e.g., ε -DP - see [KLSU19]) or exhibited significant gaps with known upper bounds (see Section 1 . 1 .
1 of [AAK21] for further discussion).
We start by stating our main result.
Theorem 3.1.
There exists a distribution D over covariance matrices Σ ∈ R d × d with I ⪯ Σ ⪯ 2 I such that, given Σ ∼ D and X ∼ N (0 , Σ) (cid:78) n , for any α = O (1) and any ( ε, δ ) -DP mechanism M : R n × d → R d × d with ε, δ ∈ [0 , 1] , and δ ≤ O (min (cid:8) 1 /n, d 2 / ( n log( n/d 2 )) (cid:9) ) that satisfies
E X,M (cid:104) ∥
M ( X ) − Σ
∥
2Σ (cid:105) ≤ α 2 , it must hold that n ≥ Ω (cid:16) d 2 αε (cid:17) .
Proof Sketch.
The outline of the proof involves (1) writing the Gaussian distribution N (0 , Σ) as an exponential family, (2) defining a distribution from which η is drawn, (3) showing that estimating the deviation of the natural parameter vector from the midpoint m reduces to covariance estimation (so lower bounds for the former also apply for the latter), (4) upper-bounding the term E (cid:2) s ⊤ Σ T s (cid:3) , and (5) applying Theorem 2.4 as described at the end of the previous section.
Step (1) is non-trivial because the standard way to write N (0 , Σ) as an exponential family involves setting η to be equal to the canonical flattening of the precision matrix (cid:0)
Σ − 1 (cid:1) ♭ .
However, we cannot use this representation, because Σ − 1 is symmetric, preventing the components of η from being generated independently, as is required by Theorem 2.4.
Thus, we consider an alternative parameterization.
This involves identifying an upper-triangular matrix U such that Σ −
=
U + U ⊤ and setting the parameter vector to be equal to 2 U ♭ .
The two parameterizations are equivalent.
Step (2) employs a variant of a construction that has appeared in [KLSU19].
The process consists of sampling the elements of the precision matrix uniformly at random from intervals of width 1 / (2 d ) . For the diagonal elements, these intervals are centered at 3 / 4 , whereas for the non-diagonal elements they are centered at the origin.
Step (3) requires us to describe an algorithm which, given oracle access to ( ε, δ ) -DP estimate (cid:98) Σ of Σ that is α -accurate in Mahalanobis norm, produces an ( ε, δ ) -DP estimate of the natural parameter vector that is O ( a ) -accurate in ℓ 2 -norm.
The idea is to calculate (cid:98) Σ − 1 , obtain a projection (cid:101) Σ − 1 of it onto the support of the output of the algorithm described in step (2), find an upper-triangular matrix (cid:101) U such that (cid:101) Σ − 1 =
U + U ⊤ and output 2 (cid:101)
U ♭ − m .
Steps (4) and (5) are mostly calculations, so we omit them from here.
■
3.2 Estimation with Respect to the Spectral Norm
Here, we prove a lower bound for covariance estimation of high-dimensional Gaussians in spectral norm under the constraint of approximate DP.
Employing a reduction-based approach from
Mahalanobis estimation to spectral estimation, we directly leverage the results of Section 3.1.
Theorem 3.2.
There exists a distribution D over covariance matrices Σ ∈ R d × d with I ⪯ Σ ⪯ 2 I such that, given Σ ∼ D and X ∼ N (0 , Σ) (cid:78) n , for any α = O (1 / √ d ) and any ( ε, δ ) -DP mechanism M : R n × d → R d × d with ε, δ ∈ [0 , 1] , and δ ≤ O (min (cid:8) 1 /n, d 2 / ( n log( n/d 2 )) (cid:9) ) that satisfies
E X,M (cid:20)(cid:13)(cid:13)(cid:13)
Σ − 12 ( M ( X ) − Σ)
Σ − 12 (cid:13)(cid:13)(cid:13) 2(cid:21) ≤ α 2 , it must hold that n ≥ Ω (cid:16) d 1 .
5 αε (cid:17) .
The proof is a reduction via the property ∥
A ∥ F ≤ √ d ∥
A ∥ 2 .
Lower Bound for Private Heavy-Tailed Mean Estimation
In this section, we state the lower bound for mean-estimation of distributions with bounded second moment with respect to the ℓ 2 norm and sketch its proof.
Theorem 4.1.
Let D be a distribution over R d with second moments bounded by 1 and unknown mean µ .
Then, for any α ≤ 1 , any ( ε, δ ) -DP mechanism with δ ≤ ε that takes X ∼ D (cid:78) n as input and outputs M ( X ) satisfying E X,M (cid:104)
∥
M ( X ) − µ ∥ 22 (cid:105) ≤ α 2 requires n ≥ Ω (cid:0) dα 2 ε (cid:1) .
Proof Sketch.
We use the DP Assouad lemma of [ASZ21].
To apply the method, we construct a family of distributions indexed by the points of the binary hypercube E d := {− 1 , 1 } d .
Define p =[-
-]
(2 α 2 ) /d and t = √ d/ ( √ 2 α ) .
Then for each v ∈ E d , we define a distribution D v over R d , such that for X ∼ D v , the following holds.
[Equation]
It can be shown that the second moment of D v is at most pt 2 = 1 .
For u, v ∈ E d , we define the loss function between D u and D v to be ℓ ( θ ( D u ) , θ ( D v ))
=
∥ θ ( D u ) − θ ( D v ) ∥ 22 .
This implies that (as defined in Lemma A.4) τ = 2 p 2 t 2 = 2 p .
On rearranging the terms in Lemma A.4, the above gives us D ≥ 0 .
04 / ( ε + δ ) .
The final task is to show that there exists a coupling ( X, Y ) between the mixture distributions p + i and p − i (as defined in Lemma A.4) with the appropriate value of D = E [ d Ham ( X, Y )] .
Here, we just construct such a coupling, but verify the bound on D in the appendix.
Let ( X, Y ) be as follows.
For every row X j in X , if X ij ̸ = 0 , then Y ij = − X ij , otherwise Y ij = X ij .
For all other coordinates k ̸ = i , Y kj = X kj .
We show that in this case, D = np .
This gives us D/p = n ≥ d/ [50( ε + δ ) α 2 )] , which proves the theorem.
■
Conclusions and Open Problems
In this work, we gave nearly tight lower bounds for private covariance estimation with respect to the
Frobenius and spectral norms under the Gaussian distribution and for mean estimation with respect to the ℓ 2 -norm under distributions with bounded second moments.
This constitutes important progress in the lower bounds literature under approximate differential privacy.
On a technical level, we significantly generalized the fingerprinting method.
An obvious question is whether the fingerprinting method can be further generalized and what are its limitations.
A second obvious question is to generalize Theorem 4.1 for higher-order moments.
Another question involves expanding upon the results of Theorems 3.1 and 3.2 to hold in the regimes α = O ( √ d ) and α = O (1) , respectively.
For Frobenius estimation, the condition α
= O (1) is a consequence of the distribution over covariance matrices that we used to prove the result, whereas for spectral estimation the condition α
= O (1 / √ d ) is an artifact of our reduction-based approach.
Especially in the latter case, it would be interesting to come up with a “direct" way to obtain the lower bound, instead of reducing from Frobenius estimation.
Finally, an interesting open question may be to obtain lower bounds for the problems considered here in the high-probability regime .
Specifically, the guarantees in our lower bounds are expressed in terms of the MSE and in Section 1.1 we sketched the argument to convert them to constant probability guarantees.
It would be interesting to come up with techniques to prove tight lower bounds when the probability of success is 1 − β for β ∈ (0 , 1) .
This was done in the context of pure differential privacy in [HKM21].
Acknowledgments
The authors would like to thank Jonathan Ullman for helpful feedback on the manuscript, Haoshu Xu for identifying a bug in the proof of Lemma 2.3 which affected how the range of δ is identified in Theorems 3.1, as well as the anonymous reviewers at NeurIPS for their comments. GK was supported by an NSERC Discovery Grant, an unrestricted gift from Google, and a University of Waterloo startup grant. AM was supported by an NSERC Discovery Grant and a David R. Cheriton
Graduate Scholarship.
VS was supported by an NSERC Discovery Grant.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendices A-G. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Preliminaries
General Notation.
We use the following notations: [ n ] := { 1 , 2 , . . . , n } , [ a ± R ] := [ a − R, a + R ] , and R + := [0 , ∞ ) .
Given any distribution D , D (cid:78) n denotes the product measure where each marginal distribution is D .
Thus, if we are given n independent samples from D , we write ( X 1 , . . .
, X n ) ∼ D (cid:78) n .
Also, depending on the context, we may use capital Latin characters like X to denote either an individual sample from a distribution or a collection of samples X :=
( X 1 , . . .
, X n ) .
To denote the j -th component of a vector, we will use either a subscript (e.g., X j , if the vector is X ) or a superscript (e.g., X ji , if the vector is X i ).
The convention used will be clear depending on the context. The unit ℓ 2 -sphere in d -dimensions that is centered at the origin is denoted by S d − 1 . Finally, we give the following definition for distributions with bounded k -th moments: Definition A.1.
Let D be a distribution over R d . We say that D has k -th moments bounded by C if E X ∼D (cid:20)(cid:12)(cid:12)(cid:12)(cid:68) X − E X
∼D [ X ] , v (cid:69)(cid:12)(cid:12)(cid:12) k (cid:21) ≤ C, ∀ v ∈ S d − 1 .
Linear Algebra Preliminaries.
For a vector v ∈ R d , v i refers to its i -th component and v − i ∈ R d −describes the vector one gets by removing the i -th component from v .
We use ⃗ 1 d to refer to the all 1 s vector in R d .
For a pair of vectors x, y ∈ R d , we write x ≤ y if and only if x i ≤ y i , ∀ i ∈ [ n ] .
Given a matrix M ∈ R d × d , we denote the element at the i -th row and the j -th column by M ij or ( M ) ij .
Also, we denote the canonical flattening operation of M into a vector by v = M ♭ ∈ R d 2 and the inverse operation by M = v # ∈ R d × d .
We denote its spectral norm by ∥ M ∥ 2 := sup v ∈ S d − 1 ∥
Mv ∥ 2 .
For symmetric matrices M , this reduces to ∥ M ∥ 2 = sup v ∈ S d − 1 (cid:12)(cid:12) v ⊤ Mv (cid:12)(cid:12) , which is known to be equal to the largest eigenvalue of M (in absolute value).
Also, we denote the trace of M by tr ( M ) : = (cid:80) i ∈ [ d ] M ii , and its Frobenius norm by ∥ M ∥ F := (cid:112) tr
( M ⊤ M ) =
(cid:115) d (cid:80) i
= d (cid:80) j =1 M 2 ij = (cid:13) (cid:13) M ♭ (cid:13) (cid:13) 2 .
For symmetric matrices, the latter is equal to the square root of the sum of the squares of its eigenvalues.
Moreover, we consider the Mahalanobis norm (with respect to a non-degenerate covariance matrix Σ ), both for vectors (cid:16) ∥ x
∥ Σ := (cid:13)(cid:13)(cid:13) Σ − 12 x (cid:13)(cid:13)(cid:13) 2 (cid:17) and matrices (cid:16) ∥ M ∥ Σ := (cid:13)(cid:13)(cid:13) Σ − 12 M Σ − 12 (cid:13)(cid:13)(cid:13) F (cid:17) .
Furthermore, we use the notation ⟨· , ·⟩ to denote inner products , either between vectors x, y ∈ R d (cid:0) ⟨ x, y ⟩ := x ⊤ y (cid:1) or between matrices A, B ∈ R d × d (cid:0) ⟨ A, B ⟩ := tr (cid:0) AB ⊤ (cid:1)(cid:1) .
The inner product of a pair of d × d matrices A, B can be reduced to an inner product between vectors in R d 2 via the identity ⟨ A, B ⟩ = (cid:10) A ♭ , B ♭ (cid:11) .
Additionally, given a pair of symmetric matrices A, B ∈ R d , we write A ⪰ B if and only if x ⊤ ( A − B ) x ≥ 0 , ∀ x ∈ R d .
Finally, given matrices A ∈ R n × m and B ∈ R k × ℓ , the Kronecker product of A with B is denoted by A ⊗ B ∈ R ( nk ) × ( mℓ ) .
More linear algebra facts appear in Appendix E.
A.1 Privacy Preliminaries
We define differential privacy and state its closure under post-processing property.
Definition A.2 (Differential Privacy (DP) [DMNS06]) .
A randomized algorithm M : X n → Y satisfies ( ε, δ ) -differential privacy ( ( ε, δ ) -DP) if for every pair of neighboring datasets X, X ′ ∈
X n (i.e., datasets that differ in exactly one entry), we have:
[Equation]
When δ = 0 , we say that M satisfies ε -differential privacy or pure differential privacy.
Lemma A.3 (Post Processing [DMNS06]) .
If M : X n → Y is ( ε, δ ) -DP, and P : Y → Z is any randomized function, then the algorithm P ◦ M is ( ε, δ ) -DP.
Finally, we discuss a recent technique from [ASZ21] for proving lower bounds for estimation problems under approximate DP.
It is a private version of the well-known Assouad Lemma [Yu97].
Let P be a family of distributions over X n , where X is the data universe and n is the number of samples.
Let θ : P → Θ be the quantity associated with the distribution that we want to estimate, and let ℓ : Θ × Θ → R + be the pseudo-metric (loss function) for estimating θ .
We define the risk of an estimator ˆ θ : X n → Θ as max p ∈P [Equation]
The minimax risk of ( ε, δ ) -DP estimators for a statistical task is defined as:
[Equation]
Let V ⊆
P be a subset of distributions indexed by the points in the hypercube E d = {− 1 , 1 } d .
Suppose there exists a τ ∈ R , such that for all u, v ∈ E d , ℓ ( θ ( p u
[Equation]
, θ ( p
[Equation]
≥
[Equation]
i =
[Equation]
{ u i
[Equation] = v i } . For every coordinate i ∈ [ d ] , define the following mixture distributions:
[Equation]
The DP Assouad’s method provides a lower bound for the minimax risk by considering the problem of distinguishing between p + i and p − i .
Lemma A.4 (DP Assouad’s Method [ASZ21]) .
For all i ∈ [ d ] , let ϕ
i : X n → {− 1 , 1 } be a binary classifier.
Then we have the following:
[Equation]
Moreover, if ∀ i ∈ [ d ] , there exists a coupling ( X, Y ) between p + i and p − i with E [ d Ham ( X, Y )] ≤ D , then:
[Equation]
A.2 Exponential Families
In this work, we make frequent use of exponential families due to their expressive power.
This gives us a unified way of dealing with multiple commonly used distributions (e.g., Gaussians, Bernoullis).
We first state the definition of exponential families.
Definition A.5 (Exponential Family) .
Let S ⊆ R d and h : S → R + , T : S → R k .
Given η ∈ R k , we say that the density function p η belongs to the exponential family E ( T, h ) if its density can be written in the form:
[Equation]
where:
[Equation]
The functions h and T are referred to as the carrier measure and the sufficient statistics of the family, respectively.
Additionally, Z is known as the log-partition function and η is the natural parameter vector .
Finally, we denote the range of natural parameters by H ⊆ R k , which is the set of values of η for which the log-partition function is well-defined ( Z ( η ) < ∞ ) .
We appeal to the following properties of exponential families.
Proposition A.6 (see [Jor10]) .
Let E ( T, h ) be an exponential family parameterized by η ∈ R k .
Then, the following hold:
• For all η ∈ H , it holds that:
[Equation]
• For all η ∈ H , it holds that:
[Equation]
• For all η ∈ H and s ∈ R k
such that η + s ∈ H , it holds that: [Equation]
For a number of facts from probability and statistics (both related and unrelated to exponential families)
[Equation]
we refer the reader to Appendix F
[Equation]
B Omitted Proofs from SectionB.1 Omitted Proofs from Section 2.
Proof of Lemma [Equation]
Without loss of generality
[Equation]
we assume that all the intervals I j are non-degenerate.
We have:
[Equation]
where ( a ) is by a direct application of Lemma 2.1.
If there exists a degenerate interval I j , we have Z j = f j ( X ) − ( η j − m j ) =
R j = 0 .
Thus, it suffices to apply the same argument as before for the non-degenerate intervals.
■
B.2 Omitted Proofs from Section 2.
This appendix is dedicated to proving Lemma 2.3 and Theorem 2.4.
To prove the former, in addition to the correlation terms defined in Section 2.1, we introduce one more correlation term.
Let X
′ i be an independently drawn sample from p η and let X ∼ i be the dataset that is the same as X with X i replaced by X ′ i .
We define:
[Equation]
This measures the effect that replacing an individual sample has on the output of the estimator.
Observe that, conditioning on η , X ∼ i and X i are independent.
This implies that
[Equation]
and
Var X ∼ i ,X i ,M [Equation] Z [Equation]
i
[Equation]
Additionally, calculating the variance of (cid:101)
Z i requires us to reason about the covariance matrix of the sufficient statistics Σ T , since the calculation involves terms of the form ( T j 1 ( X i ) − µ T,j 1 ) ( T j 2 ( X i ) − µ T,j 2 ) , j 1 , j 2 ∈ [ k ] .
The previous remarks are summarized in the following lemma.
Lemma B.1.
Conditioning on η , it holds that:
[Equation]
Proof.
We start by computing the mean.
We have:
[Equation]
where ( a ) relies on the independence of s j = (cid:104) R 2 j 4 − ( η j − m j ) 2 (cid:105)
[ M j ( X ∼ i ) − ( η j − m j )] and
T j ( X i ) − µ T,j conditioned on η .
We now proceed with the variance:
[Equation]
where ( a ) is by the second part of Proposition A.6.
■
Now, we show how to upper-bound E X,M [ Z i ] using properties of differential privacy.
We appeal to the definition of DP to relate E X,M
[ Z i ] with
E X ∼ i ,X i ,M (cid:104) (cid:101) Z i (cid:105) and E X ∼ i ,X i ,M (cid:104) (cid:101) Z 2 i (cid:105) .
The proof involves splitting Z i in its positive and negative components and applying the definition of DP to them. The main steps of the proof are standard (e.g., see [SU17b]), but we include everything for completeness.
Proof of Lemma 2.3.
Consider the random variables
Z i, + = max { Z i , 0 } and Z i, − = − min { Z i , 0 } .
These random variables are non-negative, so for any T > 0 , we have:
[Equation]
We focus our attention on the first two terms of (6).
We consider the sets S + ,X i , S − ,X i ⊆
R k :
S + ,X i :=

[Equation]
k :
[Equation]
S − ,X i :=
[Equation]

[Equation]
By a direct application of the definition of DP on S + ,X i and S − ,X i 4 for the neighboring datasets X and X ∼ i we have:
[Equation]
Under the assumption that ε ∈ [0 , 1]
[Equation]
it holds that e ε ≤ 1 + 2 ε and 1 ≥ e − ε ≥ 1 − ε ≥ 1 − 2 ε .
Combined with the above [Equation] this yields:
[Equation]
Z i
[Equation]
− t (cid:105) (cid:19) dt ≤
[Equation]
+ 2 ε ∞ (cid:90)
[Equation]
(cid:104)(cid:12)(cid:12)(cid:12) (cid:101)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
t (cid:105) dt +
[Equation]
(cid:90) (cid:18)
P X ∼ i ,X i ,M (cid:104) (cid:101)
[Equation]
i
[Equation]
(cid:105)
[Equation]
i
[Equation]
X i ,M (cid:104) (cid:101)
Z i < −
[Equation]
(cid:105)(cid:19) dt =
[Equation]
[Equation]
[Table]
+
[Equation]
[Table]
(cid:18)
[Equation] [Table] [Table] [Table]
(cid:104)
[Table]
Z
[Table]
>
[Equation]
[Table]
−
[Equation] X
[Table]
[Table] [Table] (cid:104)
[Table]
[Equation] −
[Equation]
[Table]
dt =
[Equation]
+ 2 ε
E
[Equation]
(cid:101)
Z
[Equation]
+
[Equation]
(cid:18)
[Equation]
M (cid:104)
[Table]
[Equation] [Table]
[Equation]
P
[Equation]
[Table]
[Equation] X [Equation] M (cid:104)
[Table]
Z
[Table]
< − t
[Table]
dt −
[Table]
[Equation] P
[Table]
[Equation] [Table] [Equation]
[Table]
(cid:104) (cid:101)
Z i > t (cid:105) −
P X ∼ i ,X i ,M (cid:104) (cid:101)
Z i < − t (cid:105)(cid:19) dt.
Defining (cid:101) Z i, + = max (cid:110) (cid:101)
Z i , 0 (cid:111) and (cid:101) Z i, − = − min (cid:110) (cid:101) Z i , 0 (cid:111) , we get:
T (cid:90) P X,M [ Z i > t ] dt − T (cid:90) P X,M [ Z i < − t ] dt ≤ 2 δT + 2 ε
E X ∼ i ,X i ,M (cid:104) (cid:12)(cid:12)(cid:12) (cid:101)
Z i (cid:12) (cid:12)(cid:12) (cid:105) + (cid:18)
E X ∼ i ,X i ,M (cid:104) (cid:101)
Z i, + (cid:105) − E X ∼ i ,X i ,M (cid:104) (cid:101) Z i, − (cid:105)(cid:19) − ∞ (cid:90)
T (cid:18)
P X ∼ i ,X i ,M (cid:104) (cid:101)
Z i > t (cid:105) −
P X ∼ i ,X i ,M (cid:104) (cid:101)
Z i < − t (cid:105)(cid:19) dt ≤ 2 δT + 2 ε
E X ∼ i ,X i ,M (cid:104)(cid:12)(cid:12)(cid:12) (cid:101)
Z i {+(cid:12)(cid:12)(cid:12)(cid:105) + (cid:24)(cid:24)(cid:24)(cid:24)(cid:24)(cid:24)(cid:24)(cid:58) 
+}
E X ∼ i ,X i ,M (cid:104) [Equation]
where we appealed to Lemma B.1 along the way.
The second version of Fact F.4 for X = [Equation]
(cid:101)
Z i (cid:12)(cid:12)(cid:12) , Y = 1 , and Lemma B.1 yield: [Equation]
Substituting (7) and (8) into (6), we get:
[Equation]
We now show how to bound each of the tail probabilities involving Z i and (cid:101) Z i .
For Z i , we have:
[Equation]
where ( a ) relies on the assumption | M j ( X i ) − ( η j − m j ) | ≤ R j , ∀ j ∈ [ k ] , and ( b ) uses the first version of Fact F.4 for x = ⃗ 1 k , y j = | T j ( X i ) − µ T,j | , ∀ j ∈ [ k ] .
Thus, conditioning on the dataset X , Z i is a bounded random variable supported on a 0 -centered interval.
Consequently, the worst-case upper bound we have for ∞ (cid:82)
T P X,M [ | Z i | > t ] dt is attained when all the mass of Z i is concentrated on (cid:110) ± ∥ R ∥ 3 ∞ 4 √ k ∥ T ( X i ) − µ T ∥ 2 (cid:111) for every dataset X .
Using this observation, we get:
[Equation]
Upper-bounding the term involving (cid:101) Z i in similar fashion and substituting (10) to (9), we get: [Equation]
Remark B.2. The last term in the upper bound of Lemma 2.3 involves the tail of the random variable ∥ T ( X i ) − µ T ∥ 2 . The obvious way to bound that term is:
[Equation]
Following the steps we sketched previously, this approach leads to a range for δ of the form δ ≤ (cid:101) O (cid:0) 1 n 2 (cid:1) . However, when reasoning about the cost of approximate DP for various statistical tasks, in order to claim that a full characterization has been achieved, it is generally the case that lower bounds must be shown for δ as large as (cid:101) O (cid:0) 1 n (cid:1) . To achieve this, we will apply the following strategy. First, observe that we can approximate ∥ T ( X ) − µ T ∥ 2 using an ε -net
(see Definition E.1). In particular, if N 14 is a 14 -net of S k − 1 of minimum cardinality, using Facts E.3, E.4, and a union bound, we get that:
[Equation] Now, to reason about the tail probabilities of |⟨ v, T ( X i ) − µ T ⟩| , it suffices to leverage the fact that the MGFs of univariate projections of the sufficient statistics of exponential families are well-defined (see the third part of Proposition A.6). This renders it possible to prove Chernoff-type bounds in the context that we are considering. In practice, we will not have to prove our own bounds, instead appealing to various known bounds from the literature.
We conclude this appendix with the complete proof of Theorem 2.4.
Proof of Theorem 2.4.
By Lemma 2.2 and (3), we have:
[Equation]
Additionally, by Lemma 2.3, we have:
[Equation]
Combining the two, we get the desired result [Equation]
■
C Omitted Proofs from SectionC.1 Omitted Proofs from Section 3.
The first step of the proof is to write the Gaussian N (0 , Σ) as an exponential family.
By Fact F.3, we have that the density can be written in the form:
[Equation]
Observe that the natural parameter vector η is the flattening of the precision matrix Σ − 1 .
However, we cannot use this representation, because Σ − 1 is symmetric.
Indeed, this implies that the components of η cannot be generated independently, as is required by Theorem 2.4.
For that reason, we consider an alternative parameterization.
Lemma C.1.
Let N (0 , Σ) be a d -dimensional Gaussian with unknown covariance Σ .
Then, it belongs to an exponential family with h 0 ≡ h, η 0 = 2 U ♭ , T 0 ≡ T, Z 0 ( η 0 ) =
d 2 ln (2 π
[Equation]
−
[Equation]
where U ∈ R
d × d is an upper triangular matrix with:
[Equation]
In particular, it holds that Z ( η ) =
Z 0 ( η 0 ) and η ⊤ T ( x ) =
η ⊤ 0 T 0 ( x ) .
Proof.
Observe that η # = Σ − [Equation]
=
[Equation]
0 ) ⊤= ⇒ Z ( η )
=
Z 0 ( η 0 ) .
Additionally, we have:
[Equation]
where ( a ) used the fact that x ⊤ U ⊤ x ∈ R =
⇒ x
⊤ U ⊤ x = [Equation] x ⊤ [Equation]
⊤ [Equation]
⊤ = x ⊤ Ux .
Consequently, we have p η ( x ) = h
[Equation] x ) exp [Equation] η ⊤ T
[Equation] x )
[Equation]
Z
[Equation]
η [Equation] [Equation] , verifying the equivalence of the two parameterizations.
■
We now consider the following process to generate η 0 .
Roughly speaking, modulo the parameterization issues mentioned above, the process consists of sampling the elements of the precision matrix uniformly at random from intervals of width 12 d . For the diagonal elements, these intervals are centered at 34 , whereas for the non-diagonal elements they are centered at the origin.
Similar constructions have been employed previously, e.g., in [KLSU19].
Input: d ∈ IN .
Output: A pair (Σ , η 0 ) .
1: procedure C OV S AMP ( d ) 2: for i ∈
[ d ] do 3: for j ∈ [ d ] \ [ i − 1] do 4: if i = j then 5:
Draw an independent sample (cid:0) Σ − 1 (cid:1) ii ∼ U (cid:2) 34 ± 14 d (cid:3) .
Let U ii = 12 (cid:0) Σ − 1 (cid:1) ii .
7: else 8:
Draw an independent sample (cid:0) Σ − 1 (cid:1) ij ∼ U (cid:2) ± 14 d (cid:3) .
Let (cid:0) Σ − 1 (cid:1) ji
= (cid:0) Σ − 1 (cid:1) ij .
Let U ij = (cid:0)
Σ − 1 (cid:1) ij
and U ji = 0 .
11: end if 12: end for 13: end for 14:
Let η
= 2 U ♭ .
15: return (Σ , η 0 ) .
16: end procedure
We proceed to identify some properties of the matrices generated by the above process that will help us to prove Theorem 3.1.
Lemma C.2.
Let (Σ , η 0 ) ∈
R
d × d × R d 2 be the output of Algorithm 1.
We have that:
I ⪯ Σ ⪯ [Equation] I .
The components of η 0 are supported on intervals of the form:
[Equation]
The components of η 0 are independent and it holds
that ∥ R ∥ ∞
= 1 d and ∥ R ∥ 22 = [Equation] 1 − 12 d (cid:1) ≥ 14 . 4.
The vector m ∈ R d 2 is equal to: [Equation]
d
[Equation]
otherwise .
Proof.
By a direct application of Theorem E.10, we get that the eigenvalues of Σ − 1 all lie in the interval (cid:2) 12 , 1 (cid:3) .
Thus, we have 12 I ⪯ Σ − 1 ⪯
I ⇐⇒ I ⪯ Σ ⪯ 2 I .
Observe that, the first of the three branches in the definition corresponds to the diagonal elements of η #0 , the second corresponds to those above the diagonal, and the third corresponds to those below the diagonal.
The result follows directly from this remark.
The non-constant components of η 0 are all independent draws from a uniform distribution, which immediately yields the independence property.
In addition to that, we have d nonconstant components taking values in an interval of length 12 d , and d ( d − 1)non-constant components take values in an interval of length 1 d .
This implies that ∥ R ∥ ∞
= 1 d and ∥ R ∥ [-
-]22 = d ·
[Equation] 2 + d ( d − 1) [Equation]
Observe that the non-diagonal elements of η #0 =
U are either constant and equal to 0 or take values in a 0 -centered interval.
This implies that the only non-zero elements of m are those corresponding to the diagonal elements of (cid:0) Σ − 1 (cid:1) ii .
This yields the desired result.
■
Remark C.3.
Note that Theorem 3.1 holds only for α = O (1) .
In contrast, prior fingerprinting lower bounds for mean estimation of product distributions and Gaussians gave non-trivial results in the low-accuracy regime
α = O (cid:16) √ d (cid:17) (see, e.g., the results in Section G).
In principle, it may be able to achieve similar lower bounds for covariance estimation in Frobenius norm, as the Frobenius diameter of the set of matrices that satisfy I ⪯ Σ ⪯
I is equal to √ d .
However, by the third part of the above lemma, our construction only has an upper bound on this diameter of O (1) .
This makes estimation trivial for any α which is larger, as one could output an arbitrary parameter vector within the set.
Thus, to prove a lower bound for all α = O (cid:16) √ d (cid:17) , one must consider a different construction, and it is not merely a deficiency of our analysis.
We show that, if one has an estimator for the covariance matrix Σ , then this implies an estimator for the natural parameter vector η 0 .
Therefore, a lower bound for the latter problem implies a lower bound for the former, allowing us to focus on lower bounds for estimating the natural parameter vector.
More precisely, we present an ( ε, δ ) -DP mechanism T M : R n × d → (cid:78) j ∈
[ d 2 ] I j that satisfies the guarantee
[Equation]
M assumes the existence of an ( ε, δ ) -DP mechanism M : R n × d → R d [Equation] d such that E X,M (cid:104) ∥
M ( X ) − Σ
∥
2Σ (cid:105) ≤ [Equation]
Input: X =
( X 1 , . . .
, X n ) ∼ N (0 , Σ) (cid:78) n and a mechanism M : R n × d → R d × d .
Output: T M ( X ) ∈ (cid:78) j ∈
[ d 2 ] I j .
1: procedure T M ( X ) 2:
Let (cid:98) Σ
=
M
( X ) .
3: for i ∈ [ d ] do ▷ (cid:98) Σ − 1 may not be equal to any possible Σ − 1 generated by Algorithm 1.
4: for j ∈ [ d ] do 5:
Let (cid:101) Σ − 1 ij be the projection of (cid:16)(cid:98) Σ − 1 (cid:17) ij onto the support of Σ − 1 .
6: end for 7: end for 8:
Let (cid:101) U ∈ R
d
×
d be an upper triangular matrix such that (cid:101)
Σ − 1 = (cid:101) U + (cid:101) U ⊤ .
Let T M ( X ) = 2 (cid:101)
U ♭ − m .
10: return T M ( X ) .
11: end procedure
Lemma C.4.
Let Σ ∈ R
d
×
d be a covariance generated by Algorithm 1, and X ∼ N (0 , Σ) (cid:78) n be a dataset.
If M : R n × d → R d × d is an ( ε, δ ) -DP mechanism with E X,M (cid:104)
∥
M ( X ) − Σ
∥
2Σ (cid:105) ≤ α 2 8 ≤ 132 , then T M : R n × d → (cid:78) j ∈ [ d 2 ] I j is also ( ε, δ ) -DP
and satisfies E X,
T M (cid:104) ∥ T M ( X ) − ( η 0 − m ) ∥ 22 (cid:105) ≤ α 2 ≤ 14 .
Proof.
The privacy guarantee of T M is an immediate consequence of the guarantee of M and
Lemma A.3.
Additionally, the only randomness used by T M is that used by M .
Having established that, we focus our attention on the accuracy guarantee.
We have:
[Equation]
(cid:13)(cid:101) U − U (cid:13)(cid:13) (cid:13) 2 F (cid:21) .
We remark now that:
[Equation]
where we relied on the fact that the diagonal elements of U can take only a single value.
Consequently, we get:
[Equation]
where ( a ) uses the fact that the projection we perform in Line 5 ensures that
[Equation]
Now, observe that:
[Table]
[Equation]
where ( a ) uses Fact E.6, ( b ) uses the fact that Σ − 1 ⪯ I (which holds by the first part of Lemma C.4), ( c ) uses Lemma E.7 for A
= Σ − 12 (cid:98) Σ − 12 , ( d ) is by Lemma E.8, and ( e ) is by Lemma E.9.
Substituting based on (12) into (11), we get:
[Equation]
which yields the desired result.
■
An immediate consequence of the above is the following:
Corollary C.5.
Let Σ ∈ R
d
×
d be a covariance generated by Algorithm 1, and let X ∼ N (0 , Σ) (cid:78) n .
If any ( ε, δ ) -DP mechanism T : R n × d → (cid:78) j ∈
[ d 2 ] I j with E X,T
[Equation]
[Equation]
requires at least n ≥ n η 0 samples, the same sample complexity lower bound holds for any ( ε, δ ) -DP mechanism M : R n × d → R d × d that satisfies E X,M (cid:104)
∥
M ( X ) − Σ
∥
2 Σ (cid:105) ≤ α 2 .
Having established the reduction from estimating η 0 with respect to the ℓ 2 -norm to estimating Σ with respect to the Mahalanobis norm, it remains now to apply Theorem 2.4 to the former problem.
For that reason, we work towards bounding the various quantities involved in it.
First, in accordance with Theorem 2.4, we assume the existence of an ( ε, δ ) -DP mechanism M : R n × d → (cid:78)
[Equation] I
[Equation]
such that:
[Equation]
where we appealed to the third part of Lemma C.2.
The next step to establishing our lower bound is reasoning about the quantity
[Equation]
To do that, we identify an expression for Σ T 0 for N (0 , Σ) and use a result from [DKK + 16].
By
Lemma C.1, we have T 0 ≡ T = ⇒ Σ T
= Σ T .
This yields:
[Equation] [Equation]
Observe now that, based on the definition we gave for s in Section 2 and the characteristics of the particular exponential family that we are considering here, the matrix s # ∈ R d
×
d is symmetric.
We use this observation to connect our problem with the following result:
Proposition C.6.
[Theorem 4 . 2 from [DKK + 16] ] Let S sym =
(cid:110)
M ♭ ∈ R d 2 : M = M ⊤ (cid:111) and X ∼
N (0 , Σ) .
Let M be the d 2 × d 2 matrix given by M = E (cid:104)
( X ⊗ X ) ( X ⊗ X ) ⊤ (cid:105) .
Then, as an operator on S sym , we have:
[Equation]
Using the above, we prove the following lemma:
Lemma C.7.
Let Σ ∈ R
d
×
d be a covariance matrix generated by Algorithm 1, and let X ∼
N (0 , Σ) (cid:78) n be a dataset.
Also, let M : R n × d → (cid:78) j ∈
[ d 2 ] I j be an ( ε, δ ) -DP mechanism such that
E X,M
[Equation]
[-(cid:105) ≤ 
-]
Then, it holds that:
[Equation]
Proof.
By Proposition C.6 and (14), we get that:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
where ( a ) is by the definition of the spectral norm, ( b ) is by Fact E.5, ( c ) is by the first part of Lemma C.2, ( d ) is because E X,M (cid:104)
∥
M ( X ) − ( η 0 − m ) ∥ 22 (cid:105) ≤ 8 α 2 , and the last equality is by the third part of Lemma C.2.
■
Now, it remains to reason about the terms involving the tail probabilities of ∥ T ( X i ) − µ T ∥ 2 .
Our approach closely follows the strategy sketched in Remark B.2.
The following lemma is devoted to implementing this.
Lemma C.8.
Let Σ ∈ R
d
×
d be a covariance matrix [-with I ⪯ Σ ⪯ 
 I-]generated by Algorithm 1, and X i ∼ N (0 , Σ) .
For T ≥ 14 d 2 , we have:
[Equation]
where c ≈ 0 . 036425 .
Proof.
Setting Y :
= Σ − 12 X i ∼ N (0 , I ) , we observe that:
[Equation]
where ( a ) uses Fact E.6, and ( b ) uses the assumption that Σ ⪯ 2 I and the remark that, for symmetric
PSD matrices, (cid:13)(cid:13)(cid:13) Σ 12 (cid:13)(cid:13)(cid:13) 2 = (cid:112)
∥ Σ ∥ 2 .
Thus
[Equation]
we have:
[Equation]
Working as we described in Remark B.
[Equation]
we get:
[Equation]
15) where M is a matrix with ∥ M ∥ F = 1 . We have that Y ∼ N (0 , I ) , and that M + M ⊤is symmetric, so we can apply Fact F.7 to (15). Since (cid:13)(cid:13)(cid:13) M + M ⊤(cid:13)(cid:13)(cid:13) F ≤ ∥ M ∥ F + ∥ M ⊤ ∥ F≤ 1 and (cid:13)(cid:13)(cid:13) M + M ⊤(cid:13)(cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)(cid:13) M + M ⊤(cid:13)(cid:13)(cid:13) F ≤ 1 , we get: ∞ (cid:90) T
[Equation]
[Equation]
∞ (cid:90) T exp (cid:0)
[Equation]
c min (cid:8) 16 d 4 t 2 , 4 d 2 t (cid:9)(cid:1) dt. By our assumption that on T , we get min (cid:8) 16 d 4 t 2 , 4 d 2 t (cid:9) = 4 d 2 t, ∀ t ≥ T . This leads to: ∞ (cid:90)
T
[Equation]
■
We are now ready to prove the main result of this section.
Proof of Theorem 3.1.
We assume that the process generating Σ is that of Algorithm 1.
By the first part of Lemma C.2, the condition I ⪯ Σ ⪯
I is satisfied.
We assume that there exists an ( ε, δ ) - DP mechanism M : R n × d → (cid:78) j ∈ [ d 2 ] I j that satisfies E
[Equation]
Combining the results of Lemmas C.7, and C.8, with Theorem 2.4, we get:
[Equation]
It remains to set the values of T and δ appropriately so that:
[Equation]
The first of these two conditions yields: [Equation] We set T = 12 c (cid:0) ln (3) + 12 d 2 ln (cid:0) 1 δ (cid:1)(cid:1) .
This satisfies the constraint of Lemma C.8, which says that it must hold that T ≥ 14 d 2 . Then, the above becomes:
[Equation]
The RHS of the last inequality is < 0 for d ≥ 2 . For d = 1 , we get the constraint δ ≤ e 2ln(3) − 4 ≈ 0 . Respecting this constraint, we proceed to identify a range of values for δ such that nδT ≤ 1576 . This is equivalent to:
[Equation]
The above inequality yields a constraint on δ . To ensure that ln (cid:0) 1 δ (cid:1) ≥ 0 , we need [Equation] nδ
[Equation]
{+≥ 
 ⇐⇒ δ ≤ c 288ln(3) n . We know from Lemma C.8 value of c is c ≈ 0 . 036425 . Thus, this last constraint is stricter than δ ≤ 0 .+}
For (17) to hold, we will show that it suffices to have:
[Equation]
which trivially satisfies the previous constraint. For convenience, we set ϕ : [Equation]
cd 2 288 n , so the proposed range for δ becomes δ [Equation] min [Equation]
Additionally, (17) can be equivalently written as:
[Equation] To verify our claim, we consider the following two cases: [Equation] We start by arguing that:
[Equation]
The ⇐ = direction is trivial, so we focus on the = ⇒ direction.
We assume that min
[Equation]
e
[Equation]
2ln
[Equation] [Equation]
2 but ϕ 2ln ( 1 ϕ ) >
e − 2ln(3)
d 2 .
Then, it must be the case that ϕ 2ln(3) d 2 ≤ e − 2ln(3) d 2 . Setting y :
= 1 ϕ , the above system of inequalities can be written in the form:
[Equation]
The function y ln
( y ) is increasing for y ≥ e − 1 .
We have e 2ln(3)
d 2 2ln(3) d 2 ≥ e − 1 , ∀ d ≥ 1 . Thus, for y ≥ e 2ln(3) d 2 2ln(3) d 2 , we get:
[Equation]
However, the above inequality cannot be satisfied, which leads to a contradiction.
Consequently, we have shown (19).
Since we have assumed that min [Equation] e − 2ln(3) d 2 , it must be the case that
[Equation]
We now turn our attention again to (18). We remark that, for δ ≤ e − 2ln(3) d 2 , we get ln ( 1 δ ) 2ln(3) d 2 ≥ 1 .
As a result, in order to satisfy (18), it suffices to have:
[Equation]
Observe that the function δ ln (cid:0) 1 δ (cid:1) is increasing for δ ≤ e − 2ln(3)
d 2 < e − 1 .
Combined with our previous remarks, this implies that, in order to verify (20) for the values of δ we picked, it suffices to show that it holds for δ
=
ϕ 2ln ( 1 ϕ ) .
So, we have to verify the inequality:
[Equation]
Setting z :
= ln (cid:16) 1 ϕ (cid:17) , this becomes equivalent to ln(2 z ) z ≤ 1 , which holds for all z > 0 ⇐⇒ ln (cid:16) 1 ϕ {+(cid:17) > 
+}
⇐⇒ ϕ < 1 . This last condition is satisfied by all ϕ such that [Equation]
1 ϕ ) ≤ e − 2ln(3)
d 2 , so the desired result has been established. min [Equation] e
[Equation] 2ln [Equation] 3) d 2 .
For this case, we consider two sub-cases, depending on how δ compares with e − 2ln(3) d 2 . We have: • δ ≤ e − 2ln(3)
d
As before, we argue that it suffices to have δ ln (cid:0) δ (cid:1) ≤ ϕ Since δ ln (cid:0) 1 δ (cid:1) is increasing for δ ≤ e − 2ln(3) d 2 < e − 1 , all we have to do is show that
[Equation]
[Equation]
d 2 ≤ ϕ . This is an immediate consequence of our assumption that
[Equation]
• δ > e − 2ln(3)
d 2 . This implies ln ( 1 δ ) 2ln(3) d 2 < 1 . Thus, for (18) to hold
it suffices that:
[Equation]
This is satisfied, because our proposed range for δ is δ ≤ min
[Equation]
We have established that:
[Equation]
, imply: [Equation] (3)) and 3 nδT ≤ 1 192 , while respecting all the constraints. We substitute this to (16) and get n ≥ d 2 384 αε .
Appealing to Corollary C.5 completes the proof.
■
C.2 Omitted Proofs from Section [-3.
-]3.
Proof of Theorem 3.2.
First, assume that Σ is generated by Algorithm 1 (as was assumed in all of
Section 3.1).
Additionally, assume that there exists an ( ε, δ ) -DP mechanism M : R n × d → R d × d with ε ∈ [0 , 1] and δ ≤ O (cid:18) min [Equation]
that satisfies
[Equation] ≤ α 2 and uses n = o (cid:16)
d 1 .
5 αε (cid:17) samples.
Using the standard matrix-norm inequality ∥
A ∥ F ≤ √ d ∥
A ∥ 2 , ∀ A ∈ R k × ℓ , we get: E X,M
[Equation]
By Theorem 3.1 for α → √ dα , we get that any mechanism with the characteristics of M that estimates a Gaussian covariance matrix in Mahalanobis error √ dα requires n ≥ Ω (cid:16) d 1 .
5 αε (cid:17) samples.
This leads to a contradiction, implying the desired sample complexity lower bound.
■
D Omitted Proof from Section
Proof of Theorem 4.1.
We define the following family of distributions, such that the second moment of each distribution is upper-bounded by 1 .
The loss function between the true distribution and the estimate is the squared ℓ 2 -distance between their means.
For each v ∈ E d = {− 1 , 1 } d , we define the distribution D v over R d as follows.
Let t > 0 and 0 < p < 1 .
For any X ∼ D v , we have:
[Equation]
probability
[Equation]
probability
[Equation]
Then
E X ∼D v [ X ] = ptv .
We show that the second moment of this distribution is upper-bounded by pt 2 , that is,
E
[Equation]
X − ptv, u ⟩ 2 (cid:105) ≤ pt 2 , ∀ u ∈ S d − 1 .
Note that, for any i ∈ [ d ] , we have
E [Equation]
X i − ptv i [Equation] 2 (cid:105) =
p (1 − p ) t 2 .
We omit the subscript of the expectation when the context is clear.
We have:
[Equation]
We want this to be upper-bounded by 1 : [Equation]
Next, for any u, v ∈ E d , we bound the loss function between D v 1 and D v 2 .
As mentioned before, we define ℓ ( θ ( D u ) , θ ( D v ))
=
∥ θ ( D u ) − θ ( D v ) ∥ 22 .
We have the following:
[Equation]
Using the notation in Lemma A.4, we have τ = 2 p 2 t 2 .
Using the same lemma, we have:
[Equation]
Setting R ( P , ℓ, ε, δ ) ≤ α 2 and rearranging the above, we get:
[Equation]
Now, from Equation (21), τ = 2 p 2 t 2 = 2 p .
Using this and Inequality (22), and setting p = 2 α 2 d , we have the following.
[Equation]
For i ∈ [ d ] , we define the mixture distributions p
+ i and p − i as in Lemma A.4.
We define a coupling ( X, Y ) between p + i and p − i as follows.
For every row X j in X , if X ij ̸ = 0 , then Y ij = − X ij , otherwise Y ij = X ij .
For all other coordinates k ̸
= i , Y kj = X kj .
Notice that, for all k ̸
= i , the distributions for coordinate k for both p + i and p − i are identical because all coordinates are independent.
Therefore, by setting Y kj =
X kj , these coordinates of X j and Y j have identical distributions for all j ∈ [ n ] .
The coordinate i of X j is t with probability p , and 0 with probability 1 − p .
Hence, the coordinate i of Y j is − t with probability p , and 0 with probability 1 − p .
This is true for all j ∈ [ n ] .
Since all n rows in both X (hence, in Y ) are chosen independently at random, the distribution of X is p + i and the distribution of Y is p − i .
Therefore, ( X, Y ) is a valid coupling of p + i and p − i .
Next, we have to determine the value of D = E
[ d Ham ( X, Y )] .
Note that for each j ∈ [ n ] , X j can only differ from Y j when X ij ̸ = 0 .
This can only happen with probability p .
Thus, by linearity of expectation, D = np .
Finally, using the fact that p = 2 α 2 d and t = √ d √ 2 α , and substituting this in Inequality (23), we have:
[Equation]
When ε ≥ δ , this gives us the desired lower bound.
■
E Facts from Linear Algebra and Geometry
In this appendix we collect some results from linear algebra and geometry which are referenced throughout this work.
Definition E.1 ( ε -net) . Let ( T, d ) be a metric space. Consider a subset K ⊂ T and let ε > 0 . A subset N ε ⊆
K is called an ε -net of K if the following holds:
[Equation]
Definition E.2 (Covering Number) .
The smallest possible cardinality of an ε -net of K is called the covering number of K and denoted by N ( K, d, ε ) .
Equivalently, N ( K, d, ε ) is the smallest number of closed balls with centers in K and radii ε whose union covers K .
Fact E.3 (Covering numbers of S d − 1 ) .
The covering numbers of the unit Euclidean sphere S d −satisfy the following for any ε > 0 :
[Equation]
Fact E.4 (Exercise 4 . 4 .
2 from [Ver18]) .
Let x ∈ R d
and N ε be an ε -net on S d − 1 .
It holds that:
[Equation]
Fact E.5.
Let A ∈ R n × m and B ∈ R k × ℓ .
Then, it holds that ∥ A ⊗ B ∥ 2 = ∥
A ∥ 2 ∥ B ∥ 2 .
Fact E.6.
Let M, N ∈ R d × d .
Then, ∥ MN ∥ F ≤ ∥ M ∥ 2 ∥ N ∥ F , and ∥ MN ∥ F ≤ ∥ N ∥ 2 ∥ M ∥ F .
For a proof of the first of the two inequalities, see Lemma 4 . 26 of [DKK + 16]. The proof for the second one is analogous but we have to work with the rows of M instead of the columns of N . Lemma E.7.
Let A, Σ ∈ R d
× d be non-singular matrices and let X ∈ R d × d .
Then, it holds that
[Equation]
For a proof of this statement for symmetric A (the proof does not change), we refer the reader to
Section 2 .
2 of [Li19].
Lemma E.8.
Let A,
B ∈ R d
× d be symmetric positive definite matrices.
Then, it holds that:
[Equation]
Proof.
We have that:
[Equation]
where in ( a ) we use the cyclic property of the trace in individual terms.
■
Lemma E.9.
Let Σ 1 , Σ 0 ∈ R d
×
d be a pair of symmetric positive definite matrices.
If ∥ Σ 1 − Σ[-
 ∥ Σ-] 0 ∥
Proof.
By the definition of the Mahalanobis norm, we have:
[Equation]
Similarly, we have:
[Equation]
The eigenvalues of the matrix Σ − 1 0 Σ 1 are positive.
Indeed, the previous matrix can be written as Σ
[Equation]
0 , implying that it similar to the symmetric matrix Σ − 12 0 Σ 1 Σ − 12 0 .
This last matrix is positive definite, since u ⊤ Σ − 12 0 Σ 1 Σ − 12 0 u = (cid:16) Σ − 12 0 u (cid:17) ⊤ Σ 1 (cid:16) Σ − 12 0 u (cid:17) and Σ 1 ≻ 0 .
Thus, if λ 1 , . . . , λ d > 0 are the eigenvalues of Σ − 1 0 Σ 1 , (24) and (25) can be written as:
[Equation]
Therefore, proving the desired upper bound reduces to the following maximization problem:
[Equation]
where we have omitted the constraint λ i > 0 , since we necessarily have λ i ≥ 1 − ∥ Σ 1 − Σ
∥ Σ 0 > 12 , ∀ i ∈
[ d ] .
The above problem is solvable exactly via the KKT conditions, yielding:
[Equation]
for ∥ Σ 1 − Σ
∥ Σ 0 < 12 and d ≥ 1 .
■
Theorem E.10 (Gershgorin Circle Theorem (see [Ger31])) .
Let A ∈ R d × d .
For any eigenvalue λ of
A , there exists an i ∈
[ d ] , such that:
[Equation]
F Facts from Probability & Statistics
In this appendix we collect some results from probability and statistics which are referenced throughout this work.
Fact F.1.
The Bernoulli distribution Be ( p ) is an exponential family with support S
= { 0 , 1 } and h ( x ) = 1 , η = ln (cid:16) 1 − pp (cid:17) , T ( x ) = x, Z ( η ) =
ln (1 + e η ) .
Fact F.2.
The d -dimensional Gaussian distribution with known covariance and unknown mean
N ( µ, I ) is an exponential family with support S =
R d
and h ( x ) =
(2 π ) d 2 e − ∥ x ∥ 22 2 , η = µ, T ( x ) = x
, Z ( η ) =
∥ η ∥ 22 2 .
Fact F.3.
The d -dimensional Gaussian distribution with known mean and unknown covariance N (0 , Σ) is an exponential family with support S =
R d and h ( x ) = 1 , η = (cid:0) Σ − 1 (cid:1) ♭ , T ( x ) = − 12 (cid:0) xx ⊤ (cid:1) ♭ , Z ( η ) =
d 2 ln (2 π ) − 12 ln (cid:0) det (cid:0) η # (cid:1)(cid:1) .
Fact F.4 (Cauchy-Schwarz) .
We consider the following variants of the Cauchy-Schwarz inequality: 1.
For every x, y ∈ R d , we have |⟨ x, y ⟩| ≤ ∥ x ∥ 2 ∥
y
∥ 2 .
Let X, Y be random variables over R .
Then:
[Equation]
assuming all the quantities involved are well-defined.
Fact F.5.
We denote the complementary error function by:
[Equation]
It holds that erfc ( x ) ≤ e − x 2 , ∀ x ≥ 0 .
Fact F.6 (Lemma 1 of [LM00]) .
If X ∼ χ 2 ( k ) , and β ∈ [0 , 1] , then:
[Equation]
(cid:18)β (cid:19) + 2 log (cid:18)β (cid:19)(cid:35) ≤ β, and:
[Equation]
(cid:115) k log (cid:18)β (cid:19)(cid:35) ≤ β. Equivalently, the above can be written as:
[Equation]
and:
[Equation]
Thus, if Y ∼ N (0 , I ) , then P (cid:20) ∥
Y ∥
22 ≥ d + 2 (cid:114) d log (cid:16) 1 β (cid:17) + 2 log (cid:16) 1 β (cid:17)(cid:21) ≤ β .
Fact F.7 (Gaussian Hanson-Wright Inequality - Theorem 1 and Proposition 1 from [Mos21]) .
Let
X ∈ R
d be a random vector with X ∼ N (0 , I ) .
If A ∈ R d × d is a non-zero symmetric matrix, we have:
[Equation]
where c is an absolute constant with c ≈ 0 . 036425 .
G Basic Applications: Recovering Existing Lower Bounds
We show how to use Theorem 2.4 to recover existing lower bounds from [KLSU19]: (1) mean estimation of binary product distributions, and (2) mean estimation of high-dimensional Gaussians.
Note that both classes of distributions are exponential families.
The error metric for these lower bounds would be the mean-squared-error (MSE), as opposed to constant-probability-error, as in [KLSU19], but the bounds could be converted to constant-probability bounds at the cost of a log ( d ) factor in the sample complexity (as was sketched in Section 1.1).
G.1 Mean Estimation of Binary Product Distributions
We start by stating the theorem for mean estimation of binary product distributions over { 0 , 1 } d .
Theorem G.1 (Product Distribution Mean Estimation) .
There exists a distribution D over vectors p ∈ (cid:2) 13 , 23 (cid:3) d such that, given p ∼ D and n independent samples X :=
( X 1 , . . .
, X n ) from a binary product distribution P over { 0 , 1 } d with mean p , for any α = O (cid:16) √ d (cid:17) and any ( ε, δ ) -DP mechanism
M : { 0 , 1 } n × d → (cid:2) 13 , 23 (cid:3) d with ε, δ ∈ [0 , 1] , and δ = O (cid:0) 1 n (cid:1) that satisfies E X,M (cid:104)
∥
M ( X ) − p ∥ 22 (cid:105) ≤ α 2 , it holds that n = Ω (cid:0) dαε (cid:1) .
Fact F.1 establishes that Bernoulli distributions are an exponential family.
Using that, we have that the probability mass function of binary product distributions can be written as p η = h ( x ) exp (cid:0) η ⊤ T ( x ) − Z ( η ) (cid:1) , ∀ x ∈ { 0 , 1 } d with: h ( x ) = 1 ,
T ( x ) = x, η = (cid:18) ln (cid:18) 1 − pp(cid:19) , . . . , ln (cid:18) 1 − p d p d (cid:19)(cid:19) ⊤ ,
Z ( η ) = (cid:89)
j ∈ [ d ] ln (1 + e η j ) . Our process to generate η involves drawing independently η j ∼ U [ ± ln (2)] .
We have that η j = ln (cid:16) 1 p j − 1 (cid:17) ⇐⇒ p
j = 1
+ e ηj , yielding η j ∈ [ ± ln (2)]
⇐⇒ p j ∈ (cid:2) 13 , 23 (cid:3) .
Thus, we have I j =
[ ± ln (2)] , R j = 2 ln (2) , ∀ j ∈ [ d ] = ⇒ ∥ R ∥ 22 = 4 ln 2 (2) d, ∥ R ∥ ∞
= 2 ln (2) , and m = 0 .
We now show how to reduce estimating η with an ℓ 2 -guarantee under ( ε, δ ) -DP to estimating p with an ℓ 2 -guarantee under the same constraint.
Lemma G.2.
Let p ∈ (cid:2) 13 , 23 (cid:3) d be a randomly generated vector and let X ∼ P (cid:78) n be a dataset drawn from a binary product distribution over { 0 , 1 } d with mean p .
If M : { 0 , 1 } n × d → (cid:2) 13 , 23 (cid:3) d is an ( ε, δ ) - DP mechanism satisfying E X,M (cid:104)
∥
M ( X ) − p ∥ 22 (cid:105) ≤ α 2 ≤ d 9 , then there exists a T M : { 0 , 1 } n × d → [ ± ln (2)]
d that is also ( ε, δ ) -DP
and satisfies E X,
T M (cid:104) ∥ T M ( X ) −
η ∥ 22 (cid:105) ≤ 814 α 2 .
Proof.
We define T M to be T M,j ( X ) =
ln (cid:16) 1 − M j ( X ) M j
( X ) (cid:17) , ∀ j ∈
[ d ] .
T M satisfies ( ε, δ ) -DP by
Lemma A.3 and the only randomness employed by T M is that of M .
Consider now the function g : (0 , 1) → R with g ( x ) =
ln (cid:0) 1 x − 1 (cid:1) and g ′ ( x ) = − 1 x (1 − x ) .
We have η j = g ( p j ) and
T M,j ( X ) =
g ( T j ( X )) .
Observe now that, by the Mean Value Theorem, we have for some ξ j between η j and M j ( X ) (implying that ξ j ∈ (cid:2) 13 , 23 (cid:3) ):
[Equation]
Applying this coordinate-wise, we get E X,T M (cid:104)
∥ T M ( X ) − η ∥ 22 (cid:105) ≤ 814 E X,M (cid:104)
∥
M ( X ) − p ∥ 22 (cid:105) , yielding the desired result.
■
We now present a restatement of the previous lemma from the point of view of lower bounds.
Corollary G.3.
Let p ∈ (cid:2) 13 , 23 (cid:3) d be a randomly generated vector and let X ∼ P (cid:78) n be a dataset drawn from a binary product distribution over { 0 , 1 } d with mean p .
If any ( ε, δ ) -DP mechanism
T : { 0 , 1 } n × d → [ ± ln (2)]
d satisfying E X,T (cid:104)
∥ T ( X ) − η ∥ 22 (cid:105) ≤ 814 α 2 ≤ 94 d requires at least n ≥ n η samples, the same sample complexity lower bound holds for any ( ε, δ ) -DP mechanism
M : { 0 , 1 } n × d → (cid:2) 13 , 23 (cid:3) d that satisfies E X,M (cid:104)
∥
M ( X ) − p ∥ 22 (cid:105) ≤ α 2 .
In contrast to other lower bounds proven in this work using Theorem 2.4, we do not need to study the concentration properties of ∥ T ( X i ) − µ T ∥ 2 = ∥ X i − p ∥ 2 .
Indeed, since T ( X ) = X ∈ { 0 , 1 } d , it suffices to pick a large enough threshold T > 0 so that P X i (cid:104) ∥
X i − p ∥ 2 >
[Equation]
Based on the above, we are now ready to prove the main theorem of this section. Proof of Theorem G.1. Let M : { 0 , 1 } n × d → [ ± ln (2)]
d be an ( ε, δ ) -DP mechanism with:
[Equation]
By Theorem 2.4 for product distributions with η j ∈ [ ± ln (2)] , ∀ j ∈ [ d ] , we get:
[Equation]
Observe that, since we have a product distribution and T ( x ) =
x , it holds that µ T = p and:
[Equation]
for p i ∈ (cid:2) 13 , 23 (cid:3) , ∀ i ∈
[ d ] .
Thus, we have:
[Equation]
Furthermore, we remark that ∥ X i − p ∥ 2 ≤ 23 √ d .
Thus, setting T = 4ln 3 (2) dyields:
[Equation]
Substituting (27) and (28) into (26), we get:
[Equation]
Setting δ ≤ 1 32ln(2) n results in n ≥ d 12 αε , so appealing to Corollary G.3 completes the proof.
■
G.2 Mean Estimation of High-Dimensional Gaussians
Our techniques also apply to recovering lower bounds for mean estimation of a Gaussian with known covariance.
By Fact F.2, we have that this class of distributions is an exponential family with η = µ and T ( x ) =
x
(implying that µ T = µ, Σ T = Σ ).
Therefore, we do not need to resort to a reduction-based approach (as we did with binary product distributions).
Theorem G.4 (Gaussian Mean Estimation) .
Given µ ∼ U (cid:16) [ ± 1] d (cid:17) and X ∼ N ( µ, I ) (cid:78) n , for any α = O (cid:16) √ d (cid:17) and any ( ε, δ ) -DP mechanism M : R n × d → R d with ε, δ ∈ [0 , 1] , and δ ≤
O   min  
[Equation]
    that satisfies E X,M
[Equation]
≤ α 2 , it holds that n = Ω (cid:0) d αε (cid:1) .
Before proving Theorem G.4, we start with a lemma that is concerned with upper-bounding the term involving the tail probability of ∥ T ( X i ) − µ T ∥ 2 .
The lemma does not rely on the approach of Remark B.2, instead using the concentration properties of the χ 2 -distribution (Fact F.6). Lemma G.5.
Let µ ∈ [ ± 1] d , and X i ∼ N ( µ, I ) . Assuming that T ≥ 2 d , we have: [Equation]
Proof.
Since X i ∼ N ( µ
[Equation]
I )
[Equation]
we have T ( X i ) − µ T = X i − µ . We observe that X i − µ ∼
N (0 , I ) =
⇒ ∥
X i − µ ∥ 22 ∼ χ 2 ( d ) .
By our assumption that T ≥ 2 d = ⇒ t 2 4 d ≥ T 2 4 d ≥ d , we can apply Fact F.6 with k = d .
This yields:
[Equation]
, where ( a ) is by the inequality a + b ≤ √ 2 √ a 2 + b 2 for a = z, b = z + √ d , and ( b ) is by Fact F.5.
■
We are now ready to prove the main theorem of this section. Proof of Theorem G.4. Let M : R n × d → [ ± 1] d be an ( ε, δ ) -DP mechanism with:
[Equation]
By Theorem 2.4 for Gaussians N ( µ, I ) with µ j ∈ [ ± 1] , ∀ j ∈ [ d ] , we get:
[Equation]
We have:
[Equation]
We assume that T ≥ 2 d , as is required by Lemma G.5. Later in the proof, when we pick a specific value for T , we will verify that this condition is satisfied.
Substituting based on that lemma and (30) into (29), we get:
[Equation]
It remains to set the values of T and δ appropriately so that:
[Equation]
The first of these two conditions yields: [Equation] We set T = 2 √ d [Equation]
Since δ ≤ 1 , we have ln (cid:0) 1 δ (cid:1) ≥ 0 = [Equation] T [Equation]
[Equation]
, thus satisfying the condition of Lemma G.5.
Substituting to the previous yields:
[Equation]
Setting x := (cid:113) ln (cid:0) 1 δ (cid:1) ≥ 0 , the above is a quadratic inequality with respect to x .
Depending on whether the trinomial in the LHS has any positive roots, this might imply a constraint on δ .
The discriminant is ∆ = 4 π − d , so taking cases based on d , the strictest constraint we get is δ ≤ e − π + √ 4 π − 1≈ 0 . 23 , which is for d = 1 .
Respecting this constraint, we now must identify a range of values for δ so that we have 3 nδT ≤ d 12 .
We have:
[Equation]
[Equation]
It holds that ln (cid:0) 1 δ (cid:1) + (cid:16)(cid:113) ln (cid:0) 1 δ (cid:1) + √ d (cid:17) 2 ≤ 2 (cid:16)(cid:113) ln
(cid:0) 1 δ (cid:1) + √ d (cid:17) 2 ,
so it suffices to satisfy:
[Equation]
The previous inequality imposes a condition on δ .
Indeed, to ensure that (cid:113) ln ( 1 δ ) d ≥ 0 , it is necessary that
[Equation]
, which is stronger than δ ≤ 0 . 23 .
For (32) to hold, we will show that it suffices to have:
[Equation]
 
[Equation]
which trivially satisfies the previous constraint.
For convenience, we set ϕ := √ d 144 √ 2 n , so the proposed range for δ becomes δ ≤ min
[Equation]
Additionally, (32) can be equivalently written as:
[Equation]
To verify our claim, we consider the following two cases:
min
[Equation]
≤ e − d .
We start by arguing that:
[Equation]
The ⇐ = direction is trivial, so we focus on the = ⇒ direction.
We assume that min
[Equation]
≤ e
[Equation]
but
[Equation]
ln
[Equation]
1 ϕ ) >
e − d .
Then, it must be the case that ϕ √ d ≤ e − d .
Setting y :
= 1 ϕ , the above system of inequalities can be written in the form:
[Equation]
The function y 2 ln ( y ) is increasing for y ≥ e − 12 .
We have e d √ d > e − 12 , ∀ d ≥ 1 .
Thus, for y ≥ e d √ d , we get:
[Equation]
However, this leads to a contradiction, because 3 d < 2 ln ( d ) ≤ 2 d − 2 = ⇒ d < − 2 .
Consequently, we have shown (34).
Since we have assumed that min
[Equation]
e − d , it must be the case that
[Equation]
We now turn our attention again to (33).
We remark that, for δ ≤ e − d , we get (cid:113) ln
[Equation]
d ≥ 1 .
As a result, in order to satisfy (33), it suffices to have:
[Equation]
Observe that the function δ (cid:113) ln (cid:0) 1 δ (cid:1) is increasing for δ ≤ e − d < e − 12 .
Combined with our previous remarks, this implies that, in order to verify (35) for the values of δ we picked, it suffices to show that it holds for δ = ϕ
[Equation]
ln ( 1 ϕ ) .
So, we have to verify the inequality:
[Equation]
Setting z :
{+= ln (cid:16) 1 ϕ (cid:17) , this becomes equivalent to ln ( 2 √ z ) z ≤ 3 , which holds for all z > 0 ⇐⇒ ln (cid:16) 1 ϕ (cid:17) > 
+}
⇐⇒ ϕ
< 1 .
This last condition is satisfied by all ϕ such that ϕ
[Equation]
(cid:113)
ln
1 ϕ
) ≤ e
−
d , so the desired result has been established.
min
[Equation]
> e − d .
For this case, we consider two sub-cases, depending on how δ compares with e − d .
We have:
• δ ≤ e − d .
As before, we argue that it suffices to have δ (cid:113) ln (cid:0) 1 δ (cid:1) ≤ ϕ .
Since δ (cid:113) ln (cid:0) 1 δ (cid:1) is increasing for δ ≤ e − d < e − 12 , all we have to do is show that e − d √ d ≤ ϕ .
This, however, is an immediate consequence of our assumption that min
[Equation]
ϕ
[Equation]
e − d
• δ > e − d .
This implies ln ( 1 δ ) d < 1 .
Thus,
for (33) to hold it
suffices
that
[Equation]
This is
satisfied, because our
proposed range for δ is δ ≤
min
[Equation]
We have established that:
[Equation]
imply:
[Equation]
and
[Equation]
while respecting all the constraints.
We substitute this to (31) and get n ≥ d 24 αε , which yields the desired result.
■