Private Estimation with Public Data
Abstract
We initiate the study of differentially private (DP) estimation with access to a small amount of public data.
For private estimation of d -dimensional Gaussians, we assume that the public data comes from a Gaussian that may have vanishing similarity in total variation distance with the underlying Gaussian of the private data.
We show that under the constraints of pure or concentrated DP, d + 1 public data samples are sufficient to remove any dependence on the range parameters of the private data distribution from the private sample complexity, which is known to be otherwise necessary without public data.
For separated Gaussian mixtures, we assume that the underlying public and private distributions are the same, and we consider two settings: (1) when given a dimension-independent amount of public data, the private sample complexity can be improved polynomially in terms of the number of mixture components, and any dependence on the range parameters of the distribution can be removed in the approximate DP case; (2) when given an amount of public data linear in the dimension, the private sample complexity can be made independent of range parameters even under concentrated DP, and additional improvements can be made to the overall sample complexity.
Introduction
Differential privacy (DP) [DMNS06] guarantees the privacy of every point in a dataset.
This is a strong requirement, and often gives rise to qualitatively new requirements when performing private data analysis.
For instance, for the problem of private mean estimation (under pure or concentrated
DP), the analyst must specify a guess for the unknown parameter and needs more data to get an accurate estimate depending on how good their guess is.
This cost can be prohibitive in cases where the data domain may be unfamiliar.
Fortunately, in many cases, it is natural to assume that there exists an additional public dataset.
This public dataset may vary in both size and quality.
For example, one could imagine that a fraction of users opt out of privacy considerations, giving a small set of public in-distribution data.
Alternatively, it is common to pretrain models on large amounts of public data from the web, which may be orders of magnitude larger than the private data but significantly out of distribution.
In a variety of such settings, this public data can yield dramatic theoretical and empirical improvements to utility in private data analysis (see discussion in Section 1.3).
We seek to answer the following question:
How can one take advantage of public data for private estimation?
We initiate the study of differentially private statistical estimation with a supplementary public dataset.
In particular, our goal is to understand when a small amount of public data can significantly
reduce the cost of
private
estimation. 1.1 Problem Formulation
We investigate private estimation with public data through the canonical problems of estimating
Gaussians and mixtures of Gaussians.
We focus on parameter estimation from i.i.d.
samples drawn from public and private data distributions.
We wish to design estimation algorithms that take as input both public data and private data, and are private with respect to private data.
Formally, we say a randomized algorithm A : X m × X n → Y taking as input public data (cid:101)
X ∈ X m and private data X ∈ X n is private with respect to private data if for any setting of the public data (cid:101) X , A ( (cid:101) X, · ) :
X n → Y is differentially private.
Gaussians.
Let µ, (cid:101) µ ∈
R d and positive definite Σ , (cid:101) Σ ∈
R d × d be unknown, with d TV ( N ( µ, Σ) , N ( (cid:101) µ, (cid:101) Σ))
≤ γ for some known 0 ≤ γ < 1 .
Then for 0 < α, β < 1 , given public samples from N ( (cid:101) µ, (cid:101) Σ) and private samples from N ( µ, Σ) , we would like to output ˆ µ, (cid:98) Σ such that with probability ≥ 1 − β , d TV ( N ( µ, Σ) , N (ˆ µ, (cid:98) Σ)) ≤ α .
Mixtures of Gaussians.
We represent a Gaussian mixture D by a sequence of k tuples (( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k )) , each representing the mean, covariance matrix, and mixing weight of a component.
Then for 0 < α, β < 1 , given public and private samples drawn from D , we would like to output a Gaussian mixture
(cid:98) D
= { ( (cid:98) µ 1 , (cid:98) Σ 1 , (cid:98) w 1 ) , . . .
, ( (cid:98) µ k , (cid:98) Σ k , (cid:98) w k )) } such that with probability ≥ 1 − β , for each i ∈ [ k ] , there exists a distinct j ∈ [ k ] , such that d TV ( N ( µ i , Σ i ) , N ( (cid:98) µ j , (cid:98) Σ j ))
≤ α and | w i − (cid:98)
w j | ≤ α/k .
This implies d TV ( D, (cid:98) D ) ≤ O ( α ) , and we refer to this stronger notion of component-wise estimation as ( α, β ) -learning
(Definition A.10).
1.2 Results
Gaussians.
In the case of pure and concentrated DP, exactly d + 1 public samples are sufficient to remove any dependence on “range parameters” in the private sample complexity, even when the underlying Gaussian distribution of the public data bears almost no similarity to that of the private data.
We start with following result for the scenario when the two distributions are the same.
Theorem 1.1.
For all α, β, ε > 0 and d ≥ 1 there exists a computationally efficient ε 2
-zCDP (and a computationally inefficient ε -DP) algorithm M that takes d + 1 public samples and n private samples from a Gaussian D over R d , which is private with respect to private data, such that if n = O (cid:16) d 2 α 2 + d 2 αε (cid:17) · polylog (cid:16) d, 1 β , 1 α (cid:17) , then M estimates the parameters of D to within TV error α with probability ≥ 1 − β .
Without public data, the analyst would be required to specify a bound R on the ℓ 2 norm of the mean and a bound K on the condition number of the covariance, and the private sample complexity would scale logarithmically in terms of these parameters (which we denote as range parameters ).
Thus, this cost could be arbitrarily large, or even infinite, given poor a priori knowledge of the problem.
Note that these requirements are not due to suboptimality of current private algorithms.
Packing lower bounds for estimation under pure and concentrated DP show that dependence on range parameters is an inherent cost of privacy.
However, this dependence can be entirely eliminated at the price of d +public data points, which is less public data than the non-private Θ( d 2 α 2 ) sample complexity of the problem.
For estimating Gaussians with identity or known covariance (that is, mean estimation) it turns out that just one public sample suffices to remove all dependence on range parameters.
The next natural question to ask is: what if our public data does not come from the same distribution as our private data?
Indeed, in practical settings, there may be significant domain shift between data that is publicly available and the private data of interest.
Our next result shows that we can relax this assumption: it suffices that our public data comes from another Gaussian that is at most γ < 1 away in TV distance from the underlying private data distribution.
Theorem 1.2.
For all α, β, ε > 0 , d ≥ 1 , and 0 ≤ γ < 1 , there exists a computationally efficient ε 2
-zCDP (and a computationally inefficient ε -DP) algorithm M that takes d + 1 public samples from a Gaussian (cid:101) D over R d and n private samples from a Gaussian D over R d with d TV ( D, (cid:101) D ) ≤ γ , which is private with respect to private data, such that if n = O (cid:16)
d 2 α 2 + d 2 αε (cid:17) · polylog (cid:16) d, 1 β , 1 α , 11 − γ (cid:17) , then M estimates the parameters of D to within TV error α with probability ≥ 1 − β .
The upper bound on the TV gap γ results in the sample complexity picking up an extra polylog( 11 − γ ) factor.
For instance, even when the public and private distributions have total variation distance 1 − 1 /d 100 (that is, they are almost entirely dissimilar ), we incur only an extra poly log d factor in the sample complexity.
In particular, note that the public dataset may be simultaneously quite small and significantly out of distribution with respect to the private data.
Mixtures of Gaussians.
We focus on estimation of mixtures of k separated Gaussians in d dimensions.
Specifically, we assume that, given mixture components N ( µ i , Σ i ) with respective mixing weights w i for i ∈
[ k ] (with w i ≥ w min ), then for each i ̸
= j ,
[Equation]
Private estimation of such mixtures of Gaussians (without public data) was previously studied by [KSSU19].
Their algorithms proceed in two stages: first, a clustering step to isolate the samples from each component, and an estimation step, which estimates the parameters of each individual
Gaussian using the isolated samples from each component.
Our algorithms follow a similar structure, but employ the public data at appropriate points in the procedure to enable substantial savings.
In all the results that follow, (cid:101) O hides the polylog factors of d, k , the inverse of error parameters 1 α , 1 β , and the inverse of privacy parameters 1 ε , 1 δ , 1 ρ .
We provide two flavours of results for learning mixtures of Gaussians.
First, we investigate the setting when we have a very small, dimension-independent amount of public data.
Theorem 1.3.
For all α, β, ε, δ > 0 , there exists an ( ε, δ ) -DP algorithm M that takes n private samples and m public samples from a Gaussian mixture D , satisfying (1) , which is private with respect to private data, such that if m =
O (cid:16) log( k/β ) w min (cid:17) and n = (cid:101) O (cid:16)
[Equation]
then M ( α, β ) -learns D .
Theorem 1.4.
For all α, β, ρ > 0 , there exists a ρ -zCDP algorithm M that takes n private samples and m public samples from a Gaussian mixture D satisfying (1) , which is private with respect to private data, such that if ∀ i , ∥ µ i ∥ 2 ≤ R
and I ⪯ Σ i ⪯ K I , and m =
O (cid:16) log( k/β ) w min (cid:17) and n = (cid:101)
[Equation]
(cid:16) [Equation] · polylog ( K, R ) , then M ( α, β ) -learns D .
Our results differ from the prior work of [KSSU19] in the following ways: (1) we have results for both approximate DP and zCDP, as opposed to just approximate DP in [KSSU19]; (2) the private sample complexity due to the clustering step does not involve any dependence on the range parameters; and (3) the sample complexity due to the clustering step is now O ( d 1 .
5 k 2 . 5 ) , as opposed to O ( d 1 .
5 k 9 . 06 ) , which is a significant improvement, and shows up only due to applications of private PCA under better sensitivity guarantees.
As far as the private sample complexity in the component estimation process is concerned, in the case of approximate DP, we have expressions independent of distribution parameters owing to the recent Gaussian learner by [AL21].
Finally, we investigate Gaussian mixture estimation with an amount of public data which is linear in the dimension.
Theorem 1.5.
For all α, β, ε, δ > 0 , there exists an ( ε, δ ) -DP algorithm M that takes n private samples and m public samples from a Gaussian mixture D satisfying (1) , which is private with respect to private data, such that if m =
O (cid:16) d log( k/β ) w min (cid:17) and n = (cid:101) O (cid:16) [Equation] w
[Equation]
(cid:17) , then M ( α, β ) -learns D .
Theorem 1.6.
For all α, β, ρ > 0 , there exists a ρ -zCDP algorithm M that takes n private samples and m public samples from a Gaussian mixture D satisfying (1) , which is private with respect to private data, such that if m =
O (cid:16) d log( k/β ) w min (cid:17) and n = (cid:101) O (cid:16) d 2 w min α 2 + d 2 w min α √ ρ (cid:17) , then M ( α, β ) - learns D .
Using (cid:101) O
( d/w min ) public samples establishes all the advantages that using (cid:101) O
(1 /w min ) public samples gives, but additionally improves the private sample complexity even further.
As we will describe in the technical overview, the clustering process is done entirely using the public data, and ends up removing the term due to private PCA from the private sample complexity in both approximate DP and zCDP regimes.
Additionally, in the zCDP case, we do not have any dependence on distribution range parameters even in the component estimation process because we can use results from Theorem 1.
this time.
1.3 Related Work
A large part of our work focuses on using public data to obviate the dependence on range parameters in private estimation.
Understanding these dependences without public data has been a topic of significant study.
[KV18] investigated private estimation of univariate Gaussians, showing that logarithmic dependences on the range parameters were both necessary and sufficient for pure differential privacy, but by using stability-based histograms [KKMN09; BNS16b], they could be removed under approximate differential privacy.
Similar results have been shown in the multivariate setting: while logarithmic dependences are necessary and sufficient under pure or concentrated DP [KLSU19], they can be removed under approximate DP [AAK21; KMSSU21; TCKMS21; AL21; KMV21; LKO21].
More broadly, a common theme is that the dependences which are intrinsic to pure or concentrated
DP can be eliminated by relaxing to approximate DP.
Our results demonstrate that instead of relaxing the privacy definition for all the data, one can achieve the same goal by employing a much smaller amount of public data.
Our investigation fits more broadly into a line of work employing public data for private data analysis.
Works, both theoretical and empirical, investigate the role of public data in private query release, synthetic data generation, and prediction [JE13; BNS16a; ABM19; NB20; BCMNUW20;
BMN20; LVSUW21].
For distribution-free classification with public and private data, it was shown in [ABM19] that, roughly speaking, if a hypothesis class is privately learnable with a small amount of public data ( o (1 /α ) ), it is privately learnable with no public data.
For density estimation, we show that the family of unbounded Gaussians is learnable with a small amount of public data ( α -independent), despite the fact that it is not learnable without public data.
Additionally, a number of empirical works study the efficacy of public data in private machine learning, including methods such as public pre-training [ACGMMTZ16; PCSTE19; TB21; LWAFF21;
YZCL21; LTLH22;
Yu+22], using public data to compute useful statistics about the private gradients [ZWB21; YZCL21; KRRT21; AGMRSSSTT21], or using unlabeled public data to train a student model [PAEGT17; PSMRTE18; BTGT18].
Another setting with mixed privacy guarantees is when different users may require local versus central differential privacy [AKZHL17].
Mean estimation has also been studied in this setting [ADK19].
1.4 Organization
The remainder of the paper is organized as follows.
In Section 2, we discuss our results for estimating multivariate Gaussians.
After that in Section 3, we give a technical overview of our results for estimating mixtures of Gaussians.
We present all the relevant notations and preliminaries about differential privacy, Gaussian mixtures, and techniques like PCA, along with a few useful concentration inequalities in Appendix A. In Appendix B, we provide further technical details that are missing from Section 2.
We provide all our technical results about mixtures of Gaussians from Section 3, including algorithms, theorems, and proofs, in Appendix C. Finally, in Appendix D we present some proof-of-concept numerical simulations demonstrating the effectivenss of public data for private estimation.
Estimating Gaussians
We discuss our results for estimating multivariate Gaussians with access to a limited amount of public data.
First, as a warm-up, we consider the case of estimating the mean of an identity (or known) covariance Gaussian, and give a ρ -zCDP private algorithm that uses 1 public sample.
We then generalize the high-level approach taken there to arrive at a private algorithm that uses ( d + 1) public samples for general Gaussians.
Furthermore, we show that we can relax the condition that the public and the private data come from the same distribution – the public data can come from another
Gaussian that is at most γ far away in TV distance.
2.1 Warmup: Identity Covariance Gaussians with 1 Public Sample
In this setting, we are given a single public sample (cid:101) X , along with private samples X 1 , . . .
, X n , where (cid:101) X and the X j are drawn from a d -dimensional Gaussian N ( µ, I ) independently.
Our main observation is that a single public sample is sufficient to get a “good enough” coarse estimate of µ , allowing us to apply existing private algorithms for a finer estimate.
This is captured in the following claim, which is a standard fact regarding norm concentration of Gaussian vectors (Lemma A.3).
Claim 2.1.
Suppose we get a sample (cid:101)
X ∼ N ( µ, I ) .
Then with probability at least 1 − β , ∥ µ − (cid:101) X ∥ 2 ≤ (cid:113)
[Equation]
We restate the guarantees of an existing folklore ρ -zCDP “clip-and-noise” algorithm via the Gaussian mechanism for estimating identity covariance Gaussians.
Lemma 2.2 (zCDP
Private Gaussian Mean Estimation) .
For all α, β, ρ > 0 , there exists a ρ -zCDP algorithm that takes n = (cid:101) O (cid:16) d
+log(1 /
[Equation]
[Equation]
private samples from N ( µ, I ) over
R d , where ∥ µ ∥ 2 ≤ R , and outputs an estimate (cid:98) µ ∈ R d , such that with probability at least 1 − β , d TV ( N ( µ, I ) , N ( (cid:98) µ, I )) ≤ α .
The above “clip-and-noise” mechanism simply clips all the points in the dataset to within radius λ = R + (cid:113) d + 2 (cid:112) d log( n/β ) + 2 log ( n/β ) around the origin, and outputs the noisy empirical mean of the clipped points via the Gaussian mechanism.
Claim 2.1, combined with the result above, yields a 1 -public-sample, private mean estimation algorithm via the following three-step process: (1) form shifted private dataset Y 1 , . .
, Y n by taking each Y j :=
X j − (cid:101) X ; (2) run “clip-and-noise” on Y 1 , . .
, Y n , setting R = (cid:113) d + 2 (cid:112) d log(2 /β )
+ 2 log(2 /β )
=
O ( (cid:112) d log(1 /β ))
and target failure probability β/ 2 , to obtain (cid:98) µ Y ; and (3) output (cid:98) µ := (cid:98) µ Y + (cid:101) X .
By a union bound, with probability ≥ 1 − β , the distribution of Y 1 , . . .
, Y n satisfies tight boundedness independent of the range parameters, and furthermore, succeeds in outputting a good estimate (cid:98) µ Y for µ Y :=
E Y 1 .
The last step undoes the shift to yield a good estimate (cid:98) µ for the original µ .
Privacy is preserved with respect to private data, since we are doing entry-wise pre-processing and postprocessing with public data.
For full version of this argument (which includes the ε -DP case), see
Theorem B.2.
This yields a private sample complexity that no longer depends on the range bound R , allowing us to use a fixed sample size to estimate the family of all identity covariance Gaussians over R d .
Theorem 2.3 (zCDP
Gaussian Mean Estimation with Public Data) .
For all α, β, ρ > 0 , there exists a ρ -zCDP algorithm that is private with respect to the private samples, takes 1 public sample and n = (cid:101) O (cid:16) d +log(1 /β )
α+ d log 1 . 5 (1 /β )
α √ ρ (cid:17) private samples from N ( µ, I ) over R d , and outputs an estimate (cid:98) µ ∈ R d , such that with probability at least 1 − β , d TV ( N ( µ, I ) , N ( (cid:98) µ, I )) ≤ α .
2.2 General Gaussians with d + 1 Public Samples
In this section, we discuss our main result for privately estimating Gaussians with public data.
Theorem 2.4 (Private Gaussian Estimation with Public Data) .
For all α, β, ε, ρ > 0 , there exist ρ -zCDP and ε -DP algorithms that take d + 1 public samples (cid:101) X 1 , . . .
, (cid:101)
X
d +1 and n private samples
X 1 , . . .
, X n from unknown, d -dimensional Gaussians N ( (cid:101) µ, (cid:101) Σ) and N ( µ, Σ) , respectively, with 0 ≤ d TV ( N ( µ, Σ) , N ( (cid:101) µ, (cid:101) Σ)) ≤ γ < 1 , are private with respect to the private samples, and return (cid:98) µ
X ∈ R d
and
(cid:98) Σ
X ∈ R d × d , such that d TV ( N ( (cid:98) µ X , (cid:98) Σ X ) , N ( µ, Σ)) ≤
α with probability at least 1 − β , as long as the following bounds on n hold.
For ρ -zCDP (with computational efficiency) via [KLSU19] and d + 1 public samples, n = O [Equation]
αβρ
[Equation]
d 1 .
[Equation]
polylog
[Equation]
d βρ (1 − γ ) (cid:17) √ ρ 
 .
For ε -DP (without computational efficiency) via [BKSW19] and d + 1 public samples,
[Equation]
O
[Equation]
For simplicity, the rest of this section examines the γ = 0 case, that is, when the public data and the private data distributions are the same.
Later, we address what modifications are needed for the γ >case.
Public Data Preconditioning.
We extend the same high-level idea from the identity covariance case.
We use public data to do coarse estimation, and then use that coarse estimate to transform the private data, reducing to the bounded case that can be solved using existing private algorithms (Lemma A.22).
Specifically, we use the d + 1 public samples for “public data preconditioning”.
Algorithm
Input: Public samples (cid:101) X
=
( (cid:101) X 1 , . . .
, (cid:101)
X d +1 ) .
Failure probability β > 0 .
Output: (cid:98) µ ∈ R d , (cid:98) Σ ∈ R d × d , L ∈ R , U ∈ R .
// Compute the empirical mean and covariance of (cid:101) X .
(cid:98) µ ← d +d +1 (cid:88) i = (cid:101)
X i and (cid:98) Σ ← 1 d d +1 (cid:88)
i =
( (cid:101)
[Equation] T //
Compute L and U .
L [Equation]
and U ← 9 d
[Equation]
β
Return ( (cid:98) µ, (cid:98) Σ , L, U ) .
Public-Private Gaussian Estimator.
We obtain a public-private Gaussian estimator via the following process.
(1) We use the preconditioning parameters output by Algorithm 1 to recenter, then rescale our private data X 1 , . . .
, X n .
Specifically, the transformed private data, denoted by
Y 1 , . . .
, Y n , are given by Y j := 1 √ L (cid:101)
Σ − 1 / 2 ( X j − (cid:98) µ ) .
The Y 1 , . .
, Y n are then fed as input to an existing DPGaussianEstimator (from Lemma A.22) for bounded Gaussians.
It outputs estimates (cid:98) µ Y and (cid:98) Σ Y .
For our final estimates, we output (cid:98) Σ
X :=
L (cid:98) Σ 1 / 2 Σ Y (cid:98) Σ 1 / 2 and (cid:98) µ X := √ L (cid:98) Σ 1 / 2 (cid:98) µ Y + (cid:98) µ .
See Algorithm 2 in Appendix B for an exact description.
To establish correctness, we must ensure that, with high probability over the sampling of public data, the parameters of the true distribution underlying Y 1 , . . .
, Y n indeed satisfy tight range bounds required by DPGaussianEstimator to provide the desired success guarantee with parameter-free private sample complexity.
In other words, the mean of the transformed Gaussian must lie in a known poly( d, 1 /β ) ball, and the condition number of its covariance must be poly( d, 1 /β ) .
Lemma 2.5 (Public Data Preconditioning) .
For all β > 0 , there exists an algorithm that takes d + 1 independent samples (cid:101) X 1 , . . .
, (cid:101)
X
d +1 from a Gaussian N ( µ, Σ) over R d , and outputs (cid:98) µ ∈ R d , (cid:98) Σ ∈ R d × d , L ∈ R , and U ∈ R , such that if Σ Y = 1 L (cid:98) Σ − 1 / 2 Σ (cid:98) Σ − 1 / 2 and µ Y = 1 √ L (cid:98) Σ − 1 / 2 ( µ − (cid:98) µ ) , then with probability at least 1 − β over the sampling of the data, (1) I ⪯ Σ Y ⪯ UL
I , and (2) ∥ µ Y ∥ ≤ (cid:113) UL · (cid:112) 5 log(3 /β ) , where UL = O (cid:16) d 2 log(1 /β )
β(cid:17) .
Proof.
We prove the theorem by proving the utility of Algorithm 1.
Note that the quantities L, U , as defined in Algorithm 1, satisfy U/L
= O ( d 2 log(1 /β ) /β 2 ) .
To prove part (1), where we wish to bound the eigenvalues of Σ Y , by the the properties of Loewner ordering, and since (cid:98) Σ , Σ are symmetric, we have
[Equation]
Therefore, it is sufficient to prove the final inequality.
Let (cid:98) Σ
Z :
=
Σ − 1 / 2 (cid:98) ΣΣ − 1 / 2 .
Then
[Equation]
where for each i ∈ [ d + 1] , Z i := Σ − 1 / 2 ( X i − µ ) ∼ N (0 , I ) independently and (cid:98) µ Z :=
Σ − 1 / 2 ( (cid:98) µ − µ ) = 1 d +1 (cid:80)
d +1 i =
1 Z i .
Therefore, it suffices to show
L (cid:98) Σ
Z ⪯ I and I ⪯ U (cid:98) Σ Z .
For i ∈ [ d ] , let
W i ∼ N (0 , I ) independently.
Then Cochran’s theorem (see Fact B.4) says that (cid:98) Σ Z is identically distributed to 1 d (cid:80)
[Equation]
From here, we can apply the bounds from Fact B.3 by noting that σ d ( (cid:98) Σ Z ) ∼ 1 d σ d
[ W 1 , . .
. , W d ]) 2 and σ 1 ( (cid:98) Σ Z ) ∼ 1 d σ 1 ([ W 1 , . .
. , W d ]) 2 .
With probability ≥ 1 − β/ 3 , 1 d σ d ([ W 1 , . . .
, W d ]) 2 > ( β/ 3) 2 d= ⇒ U (cid:98)
Σ
Z ⪰ I .
With probability ≥ 1 − β/ 3 , 1 d σ 1 ([ W 1 , . . .
, W d ]) 2 <
[Equation]
2 √ d + √ 2log(3 /β )
(cid:17)d = ⇒ L (cid:98) Σ
Z ⪯ I .
Taking the union bound, (1) holds with probability at least 1 − 2 β 3 .
Next, we prove the bound on µ Y in (2).
We have
[Equation]
Since (cid:98) µ Z is identically distributed to 1 √ d +1 Z 1 , by concentration of the norm for multivariate Gaussians (Lemma A.3), we have that with probability ≥ 1 − β/ 3 ,
[Equation]
d
[Equation]
From (1), we know that (cid:13)(cid:13)(cid:13) 1 L (cid:98)
[Equation]
2 ≤ UL .
Hence, (cid:13)(cid:13)(cid:13) − 1 √ L (cid:98) Σ − 1 / 2 Σ 1 / 2 (cid:13)(cid:13)(cid:13) 2 ≤ (cid:112) U/L .
This implies that
[Equation]
which implies (2).
Applying the union bound again, we have the claim.
Hence the process described above is indeed a ( d + 1) -public-sample private algorithm satisfying privacy with respect to private data X 1 , . . .
, X n and the required accuracy guarantees.
This follows from the guarantees of public data preconditioning (as stated in the aforementioned Lemma 2.5) combined with the guarantees of existing DPGaussianEstimator (stated in Lemma A.24).
See
Theorem B.5 for full details.
Again, we note that our sample complexity no longer depends on the a priori bounds on the mean and the covariance of the unknown private data distribution.
Different Public and Private Distributions.
Thus far, we have only considered the case where the public and the private data come from the same distribution.
In the more general case where the public data (cid:101) X 1 , . . .
, (cid:101)
X d +1 comes from a Gaussian (cid:101) D within TV distance γ < 1 of the private data distribution D , it can be shown that Algorithm 1 (public data preconditioning) with a slight modification will work similarly to map our unbounded problem onto a bounded one.
In particular, outputting U γ = 4 U/ (1 − γ ) 4 and L γ =
(1 − γ ) 4 L/ 4 instead of U and L leads to covariance and norm bounds K
= O ( d 2 log(1 /β )
/β
2 (1 − γ ) 8 ) and R =
O ( d log(1 /β )
/β
(1 − γ ) 4 . 5 ) on the underlying parameters of the transformed private data.
This result is the consequence of a technical
Lemma that translates the TV bound γ into a bound on Gaussian parameters.
Full details are in
Appendix B.2.
With modified Algorithm 1, we can perform the same process of: (1) transforming private data with our public data preconditioner; (2) running an existing private algorithm requiring bounded range parameters; and (3) undoing the transformation.
The guarantees of such a process are stated in
Theorem 2.4.
We pick up an extra polylog(1 / (1 − γ )) factor in our sample complexity.
Estimating Gaussian Mixtures
The general process of estimating the parameters of Gaussian mixtures involves separating the mixture components by using techniques like PCA, followed by their isolation, and finally by individual component estimation.
In other words, the components are first projected onto a low-dimensional subspace of R d , which separates the large mixture components from the rest, given that the separation in the original space is enough.
The subspace is computed with respect to the available data, that is, it is chosen to be the top k subspace of the data matrix, which preserves the original distances between the individual components, but shrinks down each component so that the intra-component distances are small but inter-component distances are large still.
This allows us to separate a group of components from the rest.
Repeating this process on each group, we can narrow down to a single
Gaussian in each group, and estimate its parameters.
This has been the high-level idea in a lot of prior work, both non-private [VW02; AM05] and private [KSSU19].
In our setting, we utilise the public data available in different settings at various points to establish these tasks, while having a low cost in the sample complexity of the private data.
Given a lower bound on the mixing weights ( w min ), we consider two settings in terms of the availability of the public data: (1) when there is very little available – (cid:101) O
(1 /w min ) ; and (2) when there is a lot more available – (cid:101) O
( d/w min ) .
The key challenge, as indicated, is accurate clustering of data.
Due to space constraints, the complete technical details for this section are deferred to Appendix C.
3.1 Public Data Sample Complexity: (cid:101) O (1 /w min )
Clustering.
This is the more challenging setting of the two because there is not enough public data for techniques such as PCA to give accurate results.
We therefore have to rely on the private data for these tasks, and use their private counterparts to get the desired results.
The entire procedure is described in detail in Algorithm 6, and its corresponding result in Theorem C.21.
Superclustering to reduce sensitivity for Private PCA:
The main goal, as in most private estimation tasks, is to reduce the amount of noise added for privacy in order to get better accuracy.
This boils down to limiting the sensitivity of the target empirical function of the data.
For this, we would like to make sure that the range of the data (to which, we scale the noise) is very tight – in our context, it means that we would like to have a ball that contains a group of mixture components, which are very close to one another (a “supercluster”), having close to the smallest possible radius.
We can limit the data to within this ball, and the sensitivity for the next step would be reduced.
To get such a ball, we use the public data, and come up with a “superclustering” algorithm (Algorithm 3, Theorem C.14), which does exactly that.
This algorithm essentially relies on the concentration properties of highdimensional Gaussians – it finds the radius of the largest Gaussian and a point (as the centre) within that Gaussian, and additively grows the ball until it stops finding more points.
This gives a ball whose radius is an O ( k ) approximation to the radius of the largest Gaussian.
We use this ball to isolate the same components in the private data, and perform private PCA to separate the components within.
Getting this tight ball is crucial to reduce the cost of private PCA in terms of the sample complexity compared to the prior work of [KSSU19].
Private PCA
: We use the well-known private PCA algorithm (Algorithm 4, Corollary C.18) that has been previously analysed in [DTTZ14; KSSU19] for this.
Partitioning in low dimensions: The next task is to partition the components in the lowdimensional subspace in a way that there is no overlap.
For this, we use an algorithm (Algorithm 5, Theorem C.20) that is similar to the “terrific ball” algorithm of [KSSU19],
and relies upon the accuracy guarantees of the private PCA step.
It uses both public and private data, and looks for a terrific ball within them.
If there is one, indicating that there is more than one component in that subset of the datasets, it partitions the datasets (and the components), and queues the partitions for further work.
Otherwise, it just adds that subset of the private dataset to the set of discovered clusters.
The whole process is repeated on all the remaining partitions.
Here, we provide some intuition about the clustering process.
To explain the implication of step (1), we state an informal version of Theorem C.14 only for the first application of Algorithm 3 when it is being run initially on the whole public dataset (of size m ).
Theorem 3.1.
There exists an algorithm, which if given m ≥ O (cid:16) ln( k/β ) w min (cid:17) samples from a Gaussian mixture D , it outputs c ∈ R d and R ∈ R , such that with probability ≥ 1 − β : (1) B R ( c ) is non-empty and contains either ≥ 1 − β/k probability mass of a Gaussian component or < β/k ; and (2) R is an
O ( k ) multiplicative approximation of the radius of the largest Gaussian within B R ( c ) .
The above result essentially says that a supercluster is captured by this algorithm in a tight ball.
Because with high probability, the points from high-dimensional Gaussians are highly concentrated,
B R ( c ) would capture all points from both public and private datasets only corresponding to the captured components.
Note that this process is performed entirely using the public dataset.
In step (2), where we perform private PCA, the radius of the ball we’re limiting the points to ( B R ( c ) ) is small, that is, R is an O ( k ) approximation of the radius of the largest mixture component contained within it.
Therefore, while computing the noisy empirical covariance, we add a relatively small amount of noise that preserves the signal of the points from that Gaussian.
This ensures that projection matrix we get preserves the inter-Gaussian distances appropriately, but significantly reduces the intra-Gaussian distances.
In other words, it preserves the distance between the mean of that largest
Gaussian from the means of the rest of Gaussians within B R ( c ) , while shrinking down all the
Gaussians.
This helps to isolate at least one component in step (3).
Step (3) relies on the following observation about Gaussians, along with the result from step (2).
In the case when there are at least two components within B R ( c ) , step (2) ensures that in the lowdimensional subspace, there is at least one Gaussian that is far from all the others.
Thus, there must be a ball that contains a large number of points, such that expanding it by a constant factor would not give us any new points (because of the available empty space), and the complement of that expanded ball would contain a large number of points.
On the other hand, if there is only one component within
B R ( c ) , then projecting it to a low-dimensional subspace would never give us such a ball.
Using this observation, Algorithm 5 uses both public and private datasets to determine whether there are one or more components in B R ( c ) , and in the first case, returns a ball that partitions the datasets in a non-trivial way that no component has points in both subsets (a “clean partition”), or in the second case, returns ⊥ .
The number of public samples is enough, such that in the first case, querying the projected public dataset for the existence of such a ball would give us one, but not enough to correctly reject in the second case, which is why we use the much larger private dataset to confirm whether the positive answer we get by querying the public dataset is correct or not.
In the end, the idea of the clustering algorithm is that in each iteration, it would either isolate both public and private points from a component or partition the datasets further in a clean way.
In either case, we move one step closer towards obtaining k clean subsets of the two datasets.
Component Estimation.
The final task is to estimate the parameters of the k isolated components (Algorithm 7, Theorem C.22).
In the ( ε, δ ) -DP setting (Theorem C.23), we simply apply the private
Gaussian learner of [AL21] to each component, which enables us to learn with no dependence in the sample complexity on the range parameters of the Gaussian itself (among other available choices, were the ( ε, δ ) -DP learners from [KMSSU21; KMV21; TCKMS21], but this one had the joint best sample complexity with [TCKMS21]).
In the ρ -zCDP setting (Theorem C.24), we use the Gaussian learner from [KLSU19], instead.
Note that in this setting, the private sample complexity does depend on the range parameters of the Gaussians, themselves.
3.2 Public Data Sample Complexity: (cid:101) O ( d/w min )
Clustering.
In the second setting, clustering is easier in terms of the tasks that need to be done privately – it can be done entirely by using the public dataset itself (Algorithm 9, Theorem C.26).
This time, the number of public samples is enough to be able to accurately do PCA on the public data, hence, perform clustering using just the public data itself.
This improves the private data sample complexity by removing the term due to private PCA altogether.
No supercustering involved: The first change in the clustering algorithm is eliminating the superclustering step.
PCA:
We essentially get the projection matrix for the top k subspace of the data using non-private PCA, and work within that subspace using the public data to further partition the two datasets (and the components).
Partitioning in low dimensions:
The algorithm for partitioning in the low-dimensional subspace (Algorithm 8, Theorem C.25) is similar to the private partitioner used in the previous case, except that there is no need for privacy in this case.
The whole process is repeated on all the remaining partitions.
Component Estimation.
Once the components are separated, as before, we individually estimate them using the partitions of the private dataset privately (Algorithm 10, Theorem C.27).
In the ( ε, δ ) -
DP setting (Theorem C.28), we again use the learner from [AL21], which ensures parameter-free private data sample complexity in this process.
In the ρ -zCDP setting though (Theorem C.29), we use our own zCDP learner from this text (Theorem 1.1), which uses the public data, as well, to ensure parameter-free private data sample complexity for this process.
Conclusion
We investigate the task of private estimation with access to a small amount of public data.
Availability of some public data is a reasonable assumption in practice, therefore, we demonstrate ways to take advantage of it.
We show that significant improvements can be made in terms of the private sample complexity for the tasks of estimating multivariate Gaussians and Gaussian mixtures, given a small amount of public data.
For the first problem, we show that even in the case where the public data distribution differs from the private data distribution significantly, we can still estimate the private data distribution using a small number of public samples, and improve the private sample complexity by removing any dependence on the range parameters of the private data distribution, which is not possible without any public data.
For the problem of estimating Gaussian mixtures under privacy constraints, our new algorithms take advantage of public data to obtain sample complexity bounds, which are significantly better than those of the best known private algorithms for this problem.
Limitations and Open Problems.
One question in the context of Gaussian estimation is whether we could obtain computationally efficient, pure DP algorithms that use public data, and match the best known private sample complexity for the problem.
Another question is whether we could polynomially improve the private sample complexity, given a much larger, but o ( d 2 ) , number of public samples.
A natural question to ask is whether our results for Gaussian estimation could be extended to mean and covariance estimation of heavy-tailed distributions.
For estimation of Gaussian mixtures, our work focuses on the case where the public data and the private data distributions are the same, so studying this problem for the case where they are different would be an interesting challenge.
Another related question is understanding other instances of clustering problems, where the results of the existing private algorithms could be improved using some additional public data.
Acknowledgments and Disclosure of Funding
AB, GK, and VS are supported by an NSERC Discovery Grant, a University of Waterloo Startup
Grant, an unrestricted gift from Google, and an unrestricted gift from Apple.
AB is supported by a
Vector Scholarship in Artificial Intelligence.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Section 2 and Appendices B and C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] No license in the repository ( https://github.com/twistedcubic/coin-press ), however we received permission from the authors to use their code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Preliminaries A.1 Notations
We define a few notations here to be used throughout the paper.
• Private data are denoted by X
=
( X 1 , . . .
X n ) , with each X j ∈ R d .
Public data is denoted by (cid:101)
X
=
( (cid:101) X 1 , . . .
, (cid:101) X m ) , with each (cid:101)
X i ∈ R d .
• We denote a ball of radius r > 0 centred around a point c ∈ R d by B r ( c ) .
• For a set S , we denote its power set by P ( S ) .
A.2 Useful Concentration Inequalities
We first state a multiplicative Chernoff bound.
Lemma A.1 (Multiplicative Chernoff) .
Let X 1 , . . .
, X m be independent Bernoulli random variables, and let X be their sum.
If p = E ( X i ) , then for 0 ≤ δ 1 ≤ 1 and δ 2 ≥ 0 ,
[Equation]
and
[Equation]
Next, we state Bernstein’s inequality.
Lemma A.2 (Bernstein’s Inequality) .
Let X 1 , . . .
, X m be independent Bernoulli random variables.
Let p =
E ( X i ) .
Then for m ≥ 5 p 2 ε 2 ln(2 /β ) and ε ≤ p/ 4 ,
[Equation]
Now, we state the Hanson-Wright inequality about quadratic forms.
Lemma A.3 (Hanson-Wright inequality [HW71]) .
Let X ∼ N (0 , I d × d ) and let A be a d × d matrix.
Then for all t > 0 , the following two bounds hold:
[Equation]
We mention an inequality that bounds the tails of a one-dimensional Gaussian N ( µ, σ 2 ) .
Lemma A.4 ( 1 -D Gaussian Concentration) .
Let Z ∼ N ( µ, σ 2 ) .
Then,
[Equation]
Next, we state a concentration inequality for 0 -mean Laplace random variables.
Lemma A.5 (Laplace Concentration) .
Let Z ∼ Lap( t ) .
Then P ( | Z | > t · ln(1 /β )) ≤ β.
We now mention an anti-concentration inequality for weighted χ 2 distributions from [ZZ18].
We adjust the constants appropriately in the following theorem as per the specifications in the aforementioned article.
Lemma A.6 (Theorem 6 from [ZZ18]) .
Let Z ∼ N (0 , 1) , a > 0 , and Y = aZ .
Then for τ ≥ a , we have the following.
[Equation]
The following are standard concentration results for the empirical mean and covariance of a set of
Gaussian vectors (see, e.g., [DKKLMS16]).
Lemma A.7.
Let X 1 , . . .
, X n be i.i.d.
samples from N (0 , I d × d ) .
Then we have that
[Equation]
where c 1 , c 2 , c 3 , c 4 > 0 are some absolute constants.
A.3 Gaussian Mixtures
Here, we provide preliminaries for the problem of parameter estimation of mixtures of Gaussians.
Let
Sym + d denote set of all d × d , symmetric, and positive semidefinite matrices.
Let G ( d ) =
{N ( µ, Σ) : µ ∈ R d ,
Σ ∈
Sym + d } be the family of d -dimensional Gaussians.
We can now define the class G ( d, k ) of mixtures of Gaussians as follows.
Definition A.8 (Gaussian Mixtures) .
The class of Gaussian k -mixtures in R d is
[Equation]
, k
[Equation]
:=
[Equation]
k >
[Equation]
k
[Equation]
i
[Equation]
w i
[Equation]
(cid:41) .
We can specify a Gaussian mixture by a set of k tuples as: { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } , where each tuple represents the mean, covariance matrix, and mixing weight of one of its components.
Additionally, for each i , we refer to σ 2 i = ∥
Σ i ∥ 2 as the maximum directional variance of component i .
We assume that the mixing weight of each component is lower bounded by w min .
We impose a separation condition for mixtures of Gaussians to be able to learn them.
Definition A.9 (Separated Mixtures) .
For s > 0 , a Gaussian mixtures D ∈ G ( d, k ) is s -separated if
[Equation]
We denote the family of separated Gaussian mixtures by G ( d, k, s ) .
Now, we define what it means to “learn” a Gaussian mixture in our setting.
Definition
A.
( ( α, β ) -Learning) .
Let
D ∈
G ( d, k ) be parameterized by { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } .
We say that an algorithm ( α, β ) -learns D , if on being given sample-access to D , it outputs with probablity at least 1 − β a distribution (cid:98) D ∈ G ( d, k ) parameterised by { ( (cid:98) µ 1 , (cid:98) Σ 1 , (cid:98) w 1 ) , . . .
, ( (cid:98) µ k , (cid:98) Σ k , (cid:98) w k ) } , such that there exists a permutation π : [ k ] →
[ k ] , for which the following conditions hold.
For all 1 ≤ i ≤ k , d TV ( N ( µ i , Σ i ) , N ( (cid:98) µ π ( i ) , (cid:98) Σ
π ( i ) ))
≤ O ( α ) .
For all 1 ≤ i ≤ k , (cid:12)(cid:12) w i − (cid:98) w π ( i ) (cid:12)(cid:12) ≤ O (cid:0) αk (cid:1) .
Note that the above two conditions together imply that d TV ( D , (cid:98) D ) ≤ α .
Finally, we define the “median radius” of a Gaussian.
Definition A.11.
Let G :=
N ( µ, Σ) be a d -dimensional Gaussian, and R > 0 .
Then R is called the median radius of G , if the G -measure of B R ( µ ) is exactly 1 / 2 .
The following is an anti-concentration lemma from [AK01] about Gaussians.
It lower bounds the distance of a sample from a Gaussian from any arbitrary point in space.
Lemma A.12 (Lemma 6 from [AK01]) .
Let G :=
N ( µ, Σ) be a Gaussian in R d with median radius
R , and let σ 2 = ∥ Σ ∥ .
Suppose z ∈ R d is an arbitrary point, and x ∼ G .
Then for all t ≥ 1 , with probability at least 1 − 2 e − t ,
[Equation]
A.4 Robustness of PCA to Noise
Principal Component Analysis (PCA) is one of the main tools in learning mixtures of Gaussians.
It is common to project the entire data onto the top- k principal directions (a subspace, which would approximately contain all the means of the k components).
Ideally, PCA should eliminate directions that are not so useful, whilst maintaining the distances among the Gaussian components.
This would allow us to cluster the data in low dimensions easily based on the most useful directions, while not having to worry about the rest of the directions that do not give us useful information for this process.
For the purpose of doing PCA under differential privacy, we would like to have results for PCA when there is noise involved in the process of obtaining DP guarantees.
Let X ∈ R n × d be the dataset obtained from the mixture of Gaussians in question.
Suppose A ∈ R n × d , such that for each i , A
i is the true mean of the Gaussian from which X i has been sampled.
Let n j denote the number of points in X belonging to component j .
The following lemma gives guarantees for PCA when we have a noisy approximation to the top- k subspace of the dataset.
Lemma A.13 (Lemma 3.1 from [KSSU19]) .
Let X ∈ R n × d be a collection of n datapoints from k clusters each centered at µ 1 , µ 2 , ..., µ k .
Let A ∈ R n × d be the corresponding matrix of (unknown) centers (for each j we place the center µ c ( j ) with c ( j ) denoting the clustering point
X j belongs to).
Let Π V k ∈ R
d
× d denote the k -PCA projection of X ’s rows.
Let Π U ∈
R
d × d be a projection such that for some bound B ≥
it holds that ∥ X T X − ( X Π U ) T ( X Π U )
∥ 2 ≤ ∥ X T X − ( X Π V k ) T ( X Π V k ) ∥ 2 + B .
Denote ¯ µ i as the empirical mean of all points in cluster i and denote ˆ µ i as the projection of the empirical mean ˆ µ i
=
Π
U ¯ µ i .
Then
[Equation]
The next lemma bounds the singular values of a matrix X that is sampled from a mixture of Gaussians, but centred around A .
Lemma A.14 (Lemma 3.2 from [KSSU19]) .
Let X ∈ R n × d be a sample from D ∈ G ( d, k ) , and let
A ∈ R n × d be the matrix where each row i is the (unknown) mean of the Gaussian from which X i was sampled.
For each i , let σ
i denote the maximum directional variance of component i , and w i denote its mixing weight.
Define σ 2 = max i { σ 2 i } and w min =
min i { w i } .
If
[Equation]
where ξ 1 , ξ 2 are universal constants, then with probability at least 1 − β ,
[Equation]
A.5 Privacy Preliminaries
We start with different definitions of differential privacy.
Definition A.15 (Differential Privacy (DP) [DMNS06]) .
A randomized algorithm M : X n → Y satisfies ( ε, δ ) -differential privacy ( ( ε, δ ) -DP) if for every pair of neighboring datasets X, X ′ ∈
X
n (i.e., datasets that differ in exactly one entry),
[Equation]
When δ = 0 , we say that M satisfies ε -differential privacy or pure differential privacy.
Definition A.16 (Concentrated Differential Privacy (zCDP)
[BS16]) .
A randomized algorithm
M : X n → Y satisfies ρ -zCDP if for every pair of neighboring datasets X, X ′ ∈
X n ,
[Equation]
where D α ( M ( X ) || M ( X ′ )) is the α -Rényi divergence between M ( X ) and M ( X ′ ) .
Note that ( ε, 0) -DP implies ε 2 2 -zCDP, which implies ( ε (cid:112) log(1 /δ ) , δ ) -DP for every δ >
[BS16].
Now, we define the notion of “public-private” algorithms that take public data samples and private data samples as input, and guarantee differential privacy with respect to the private data.
Definition A.17 (Public-Private Algorithms) .
Let (cid:101)
X be the domain of public data and X be the domain of private data.
A randomized algorithm M : X n × (cid:101)
X m → Y taking in public and private data satisfies ( ε, δ ) -DP
(or ρ -zCDP) with respect to the private data if for any public dataset (cid:101) X ∈ (cid:101)
X m , the resulting randomized algorithm M ( (cid:101) X, · ) :
X n → Y is ( ε, δ ) -DP (or ρ -zCDP, respectively).
These definitions of DP are closed under post-processing, and can be composed with graceful degradation of the privacy parameters.
Lemma A.18 (Post-Processing [DMNS06; BS16]) .
If M : X n → Y is ( ε, δ ) -DP, and P : Y → Z is any randomized function, then the algorithm P ◦ M is ( ε, δ ) -DP.
Similarly if M is ρ -zCDP then the algorithm P ◦ M is ρ -zCDP.
Lemma A.19 (Composition of DP [DMNS06; DRV10; BS16]) .
If M is an adaptive composition of differentially private algorithms M 1 , . .
, M T , then the following all hold:
If M 1 , . . . , M T are ( ε 1 , δ 1 ) , . . .
, ( ε T , δ T ) -DP then M is ( ε, δ ) -DP for ε =
(cid:88) t ε t and δ = (cid:88) t δ t .
If M 1 , . . . , M T are ( ε 0 , δ 1 ) , . . .
, ( ε 0 , δ T ) -DP for some ε 0 ≤ 1 , then for every δ 0 > 0 , M is ( ε, δ ) -DP for ε = ε 0 (cid:112) 6 T log(1 /δ 0 ) and δ =
δ 0 + (cid:88) t δ t 3.
If M 1 , . . . , M T are ρ 1 , . .
, ρ T -zCDP then M is ρ -zCDP for ρ = (cid:80) t ρ t .
A.5.1 Known Differentially Private Mechanisms
We state standard results on achieving differential privacy via noise addition proportional to the sensitivity [DMNS06].
Definition A.20 (Sensitivity) .
Let f : X n → R d be a function, its ℓ 1 -sensitivity and ℓ 2 -sensitivity are
[Equation]
respectively.
Here, X ∼ X ′ denotes that X and X ′ are neighboring datasets (i.e., those that differ in exactly one entry).
For functions with bounded ℓ 1 -sensitivity, we can achieve ε -DP by adding noise from a Laplace distribution proportional to ℓ 1 -sensitivity.
For functions taking values in R d for large d it is more useful to add noise from a Gaussian distribution proportional to the ℓ 2 -sensitivity, to get ( ε, δ ) -DP
and ρ -zCDP.
Lemma A.21 (Laplace Mechanism) .
Let f : X n → R d be a function with ℓ 1 -sensitivity ∆ f, 1 .
Then the Laplace mechanism
[Equation]
satisfies ε -DP.
Lemma A.22 (Gaussian Mechanism) .
Let f : X n → R d be a function with ℓ 2 -sensitivity ∆ f, 2 .
Then the Gaussian mechanism
[Equation]
satisfies ( ε, δ ) -DP.
Similarly, the Gaussian mechanism
[Equation]
satisfies ρ
-zCDP.
Next, we mention a very basic pure DP algorithm that computes the cardinality of a dataset as a simple application of Lemma A.21.
Lemma A.23 ( PCount ) .
Let X =
( X 1 , . . .
, X n ) be a set of points from some data universe χ .
Then for all ε > 0 and 0 < β < 1 , there exists an ε -DP mechanism ( PCount : χ ∗ → R ) that on input X outputs n ′ , such that with probability at least 1 − β , | n − n ′ | ≤ ln(1 /β )
ε .
Proof.
The algorithm is just the following.
Step 1: sample γ ∼ Lap (cid:0) 1 ε (cid:1) .
Step 2: output | X | + γ .
Since the sensitivity of | X | is 1 , by Lemma A.21, PCount is ε -DP.
Next, by the guarantees of Lemma A.5, | γ | ≤ ln(1 /β )
ε with probability at least 1 − β .
Now, we mention three results from prior work for learning high-dimensional Gaussians under approximate DP, z CDP, and pure DP constraints, respectively.
Lemma A.24 ( DPGaussianEstimator ) .
Let Σ ∈ R
d
× d be symmetric and positive-definite, and µ ∈ R d .
For all 0 < α, β < 1 , if given n GE independent samples from N ( µ, Σ) , then the following algorithms output a symmetric and positive-definite (cid:98) Σ ∈ R d × d , and a vector (cid:98) µ ∈ R d , such that with probability at least 1 − β , d TV ( N ( µ, Σ) , N ( (cid:98) µ, (cid:98) Σ)) ≤ α .
For the ( ε, δ ) -DP Gaussian learner from [AL21],
[Equation]
For the ρ -zCDP Gaussian learner from [KLSU19], for I ⪯ Σ ⪯ K I and ∥ µ ∥ ≤ R (where K ≥ 1 and R > 0 ),
[Equation]
For the (computationally inefficient) ε -DP Gaussian learner from [BKSW19], for I ⪯ Σ ⪯ K I and ∥ µ ∥ ≤ R (where K ≥ 1 and R > 0 ),
[Equation]
B Estimating Gaussians
Here we present missing details from Section 2.
B.1 Same Public and Private Distributions B.1.1 Unknown Mean, Identity Covariance with 1 Public Sample
In this setting, we are given a single public sample (cid:101) X , along with private samples X 1 , . . .
, X n , where the (cid:101) X and the X j are drawn from a d -dimensional Gaussian N ( µ, I ) independently.
Recall our observation from Section 2: a single public sample is sufficient to get a “good enough” coarse estimate of µ , allowing us to apply existing private algorithms for a finer estimate.
This was captured in Claim 2.1.
We restate existing pure and concentrated DP algorithms for Gaussian mean estimation below.
Lemma B.1 (Known Private Mean Estimators for Identity Covariance Gaussians) .
For all α, β, ε, ρ > 0 , there exist ε -DP
and ρ -zCDP algorithms that take n > 0 samples from N ( µ, I ) over R d , where ∥ µ ∥ 2 ≤ R , and output estimate (cid:98) µ ∈ R d , such that with probability at least 1 − β , d TV ( N ( µ, I ) , N ( (cid:98) µ, I ))
≤ α , as long as the following bounds on n hold.
For the ρ -zCDP “clip-and-noise” algorithm via the Gaussian Mechanism,
[Equation]
For the (inefficient) ε
-DP algorithm from [BKSW19],
[Equation]
For the ε -DP algorithm from [HKM22],
[Equation]
In the above, the “clip-and-noise” mechanism simply clips all the points in the dataset to within radius λ = R + (cid:113) d + 2 (cid:112) d log( n/β ) + 2 log ( n/β ) around the origin, and outputs the noisy empirical mean of the clipped points via the Gaussian mechanism.
Note that the ℓ 2 sensitivity of the empirical mean is now 2 λn , therefore, the magnitude of the noise vector added to the empirical mean is at most (cid:101) O (cid:16) √ dλ √ ρn (cid:17) with high probability, which is less than α when n ≥ (cid:101) O (cid:16) √ dλ √ ρα (cid:17) .
This is ρ -zCDP from
Lemma A.22.
Also, in part (3) of the above lemma, the third term of the sample complexity arises when the algorithm tries to find a coarse estimate of the mean, that is, when it attempts to get an estimate that lies within O ( √ d ) of the true mean (Theorem 5.1 of [HKM22]).
If we already have such a coarse estimate with us (which we will), then that term would disappear, and we would simply have to apply
Theorem 6.1 from [HKM22] to get the first term.
Claim 2.1 combined with the results above, yields 1 -public-sample, private mean estimation algorithms.
Theorem B.2 (Public-Private Gaussian Mean Estimation) .
For all α, β, ε, ρ > 0 , there exist ε -DP
and ρ -zCDP algorithms that take n > 0 private samples X
=
( X 1 , . . .
, X n ) and 1 public sample (cid:101) X from N ( µ, I ) over R d , are private with respect to the private samples X , and output estimate (cid:98) µ ∈ R d , such that with probability at least 1 − β , d TV ( N ( µ, I ) , N ( (cid:98) µ, I ))
≤ α , as long as the following bounds on n hold.
For the ρ -zCDP “clip-and-noise” algorithm via the Gaussian Mechanism and 1 public sample,
[Equation]
For the (inefficient) ε
-DP algorithm via [BKSW19] and 1 public sample,
[Equation]
For the (efficient) ε -DP algorithm via [HKM22] and 1 public sample,
[Equation]
Proof.
For concreteness, we look at (1), however an analogous argument suffices for (2) and (3).
Say we draw our one public sample (cid:101) X , and form a shifted private dataset Y 1 , . . .
, Y n , each Y j
= X j − (cid:101) X .
Define λ = (cid:113) d + 2 (cid:112) d log(2 /β )
+ 2 log (2 /β ) and µ Y :=
E ( Y 1 ) = µ − (cid:101) X .
By Claim 2.1, we have that with probability ≥ 1 − β/ 2 over the sampling of (cid:101) X , ∥ µ Y ∥ 2 ≤ λ .
Hence, we set R = λ and target failure probability β/ 2 , and run the zCDP mean estimation algorithm from Lemma B.1 on
Y 1 , . . .
, Y n .
Suppose the output of the algorithm is (cid:98) µ Y .
Then we return (cid:98) µ = (cid:98) µ Y + (cid:101) X .
By union bound, with probability ≥ 1 − β , we have that our private data distribution satisfies the range requirements for our private algorithm’s guarantees to hold and that our private algorithm succeeds.
In this case, we have ∥ µ − (cid:98) µ ∥ 2 = ∥ ( µ − (cid:101) X ) − ( (cid:98) µ − (cid:101) X ) ∥ 2 = ∥ µ Y − (cid:98) µ Y ∥ 2 , so d TV ( N ( (cid:98) µ, I ) , N ( µ, I ))
=
d TV ( N ( (cid:98) µ Y , I ) , N ( µ Y , I )) ≤ α .
Plugging in R = λ and target failure probability β/ 2 into the algorithm’s sample complexity in Lemma B.1 gives us the desired result.
As mentioned earlier, if we are to use part (3) of Lemma B.1, then the third term of the sample complexity would disappear because the public sample already gives us an O ( √ d ) estimate of µ .
Notice that (cid:98) µ Y is ρ -zCDP with respect to Y 1 , . .
, Y n , which implies that (cid:98) µ is ρ -zCDP with respect to private data X 1 , . . .
, X n .
To see why, note that for any fixed (cid:101) X , the private algorithm from
Lemma B.1 is robust to arbitrary change in Y j , therefore, to any change in X j because each X j maps to exactly one Y j .
By the post-processing guarantee of zCDP
(Lemma A.18), (cid:98) µ is also private with respect to X .
We remark that the above 1-public-sample private mean estimators can also be used to estimate the mean of a Gaussian with arbitrary known covariance, by reducing to the identity covariance case via rescaling with the known covariance.
B.1.2 Unknown Mean and Covariance with d + 1 Public Samples
In this setting, we are given d + 1 public samples (cid:101) X 1 , . . .
, (cid:101)
X
d +1 and n private samples X 1 , . .
, X n , where all the (cid:101)
X i and X j are drawn from an unknown, d -dimensional Gaussian N ( µ, Σ) independently.
Public data is used to transform private samples via “public data preconditioning” (Algorithm 1, as discussed in Section 2), which reduces the estimation problem to the bounded case which can be solved using the existing private algorithms.
Algorithm 2 gives the precise description of how we can use Algorithm 1 and existing private
Gaussian estimators to obtain a public-private Gaussian estimator.
Public samples (cid:101) X
=
( (cid:101) X 1 , . . .
, (cid:101)
X d +1 )
Private samples X
=
( X
Error tolerance α > failure β >
⊂ R .
Output
// Precondition the private data using the public dataset.
( (cid:98) µ, (cid:98) Σ , L, U )
← PubPreconditioner β 2 ( (cid:101) X ) .
For j ∈ [ n ]
Set Y
j ← 1 √ L (cid:98) Σ − 1 / 2 ( X j − (cid:98) µ ) .
// Set range parameters for private applications.
R = (cid:112) U/L · (cid:112) log(6 /β ) and K =
U/L // Estimate Gaussian using private data and private algorithm.
Set Y ← ( Y 1 , . . .
, Y n ) and ( (cid:98) µ Y , (cid:98) Σ Y ) ← DPGaussianEstimator α, β 2 , PrivParams ,R,K ( Y ) .
Set (cid:98) µ X ← √ L (cid:98) Σ 1 / 2 (cid:98) µ Y + (cid:98) µ and (cid:98) Σ
X ← L (cid:98) Σ 1 / 2 (cid:98) Σ Y (cid:98) Σ 1 /
Return (cid:98) µ X , (cid:98) Σ X .
The proof of the correctness of Algorithm 2 relies on proof of correctness for public data preconditioning, which was given in Section 2.
The following facts were used for the proof in that section.
Fact B.3 (Singular Values of Gaussian Matrices [SST06]) .
Let Z ∈ R
d
×
d be a matrix with each
Z ij ∼ N (0 , 1) independently.
Denote by σ d ( Z ) the smallest singular value of Z , and by σ 1 ( Z ) its largest singular value.
Then we have the following.
[Equation]
Fact B.4 (Properties of Wishart Distribution 3 ) .
Let Z 1 , . . .
Z m +1 , W 1 , . .
, W m be chosen i.i.d.
from N (0 , I ) over R d .
Suppose (cid:98) µ = 1 m + m +1 (cid:80) i =1 Z i .
Then
[Equation]
i =
[Equation]
In other words, the two quantities are identically distributed according to the scaled, d -dimensional
Wishart distribution with m degrees of freedom.
Below is the same-distribution version of Theorem 2.4, which employs Lemma 2.5 from Section 2.
Theorem B.5 (Public-Private Gaussian Estimation) .
For all α, β, ε, ρ > 0 , there exist ε -DP
and ρ -zCDP algorithms that take d +1 public samples (cid:101)
X
=
( (cid:101) X 1 , . . .
, (cid:101)
X d +1 ) and n private samples X
=
( X 1 , . . .
, X n ) from an unknown, d -dimensional Gaussian N ( µ, Σ) , and are private with respect to the private samples, and return (cid:98) µ X ∈ R d
and
(cid:98) Σ
X ∈ R d × d , such that d TV ( N ( (cid:98) µ X , (cid:98) Σ X ) , N ( µ, Σ)) ≤
α with probability at least 1 − β , as long as the following bounds on n hold.
For ρ -zCDP (with computational efficiency) via [KLSU19] and d + 1 public samples,
[Equation]
For ε -DP (without computational efficiency) via [BKSW19] and d + 1 public samples,
[Equation]
Proof.
We prove the privacy and the utility guarantees for Algorithm 2.
In this proof, we will work in the ρ -zCDP regime (where, PrivParams = { ρ } ) using the Gaussian estimator of [KLSU19]
(the second algorithm in Lemma A.24), and the analogous guarantees in the ε -DP case (where
PrivParams = { ε } ) would follow by the same argument (but by using the third algorithm from
Lemma A.24, instead).
We start by proving the utility guarantees first.
Applying our public data preconditioner (Algorithmwith public data (cid:101) X as input) on X with target failure probability β/ 2 yields Y 1 , . . .
, Y n , such that for each j ∈ [ n ] , Y j ∼ N ( µ Y , Σ Y ) , where µ Y and Σ Y are quantites as defined in Lemma 2.5.
Then with probability ≥ 1 − β/ 2 over the sampling of public data, we have I ⪯ Σ Y ⪯ UL
I and ∥ µ Y ∥ 2 ≤ (cid:112) U/L · (cid:112) 5 log(6 /β )
(Lemma 2.5).
Hence, we can set K = U/L
= O ( d 2 log(1 /β ) /β 2 ) , and R = (cid:112) U/L · (cid:
) 5 log(6 /β )
=
O ( d log(1 /β ) /β ) , and run the ρ -zCDP Gaussian estimator on
Y 1 , . . .
, Y n with target failure probability β/ 2 .
We obtain our private sample complexity, which is now independent of the range parameters of the underlying distribution, by plugging in these values into the private Gaussian estimator’s sample complexity.
Under these parameter settings and sample complexity, by Lemma A.24 and the union bound, we have that with probability ≥ 1 − β , the algorithm succeeds in outputting (cid:98) µ Y and (cid:98) Σ Y , such that d TV ( N ( (cid:98) µ Y , (cid:98) Σ Y ) , N ( µ Y , Σ Y ))
≤ α .
We output the estimates (cid:98) Σ X : − L (cid:98) Σ 1 / 2 (cid:98) Σ Y (cid:98) Σ 1 / 2 and (cid:98) µ X := √ L (cid:98) Σ 1 / 2 (cid:98) µ Y + (cid:98) µ .
Denoting A :
= 1 L (cid:98) Σ − 1 , by the properties of the Mahalanobis norm, ∥ · ∥ Σ :
[Equation]
which implies that d TV ( N ( (cid:98) µ X , (cid:98) Σ X ) , N ( µ, Σ))
=
d TV ( N ( (cid:98) µ Y , (cid:98) Σ Y ) , N ( µ Y , Σ Y )) ≤ α .
To argue about privacy, note that releasing (cid:98) µ X and (cid:98)
Σ
X is ρ -zCDP with respect to Y , since it is postprocessing (Lemma A.18) of the output ( (cid:98) µ Y , (cid:98) Σ Y ) of a ρ -zCDP algorithm using public information.
To argue about the ρ -zCDP of our algorithm with respect to the private dataset X , note that for any fixed public dataset (cid:101) X , the application of DPGaussianEstimator from Lemma A.24 is robust to changing one sample Y j arbitrarily.
Since each X j maps to exactly one Y j , DPGaussianEstimator is robust against changing any X j arbitrarily, as well.
This gives us the final privacy guarantee with respect to X .
B.2 Different Public Data and Private Data Distributions
In this section, we give results for the scenario, where the public data does not come from the same distribution as that of the private data.
Specifically, we show that in the case, where our public data comes from another Gaussian within TV distance γ of our private data distribution, a slight modification of Algorithm 1 (public data preconditioning) will also work to reduce our unbounded private estimation problem to a bounded one.
B.2.1 Technical Lemmata
We start by stating results about different distance metrics for distributions.
Let d H ( · , · ) denote the Hellinger distance between two distributions, and let d TV ( · , · ) denote their total variation (TV) distance.
We start with a known fact about the relation between d H ( · , · ) 2 and d TV ( · , · ) .
Lemma B.6 (Hellinger Distance vs. TV Distance) .
Let P, Q be distributions over R d .
Then d H ( P, Q ) 2 ≤ d TV ( P, Q ) .
Next, we state the expression for Hellinger distance between two univariate Gaussians.
Lemma B.7 (Hellinger Distance for Gaussians) .
Let G 1 ≡ N ( µ 1 , σ 21 ) , G 2 ≡ N ( µ 2 , σ 22 ) be Gaussians over R .
Then
[Equation]
Now, we lower-bound the Hellinger distance between two univariate Gaussians.
Lemma B.8 (Hellinger Distance Lower Bounds) .
Let G 1 ≡ N ( µ 1 , σ 21 ) , G 2 ≡ N ( µ 2 , σ 22 ) be Gaussians over R .
Suppose σ max = max { σ 1 , σ 2 } .
Then d H ( G 1 , G 2 )≥ d H ( N ( µ 1 , σ 2max ) , N ( µ 2 , σ 2max )) 2 and d H ( G 1 , G 2 ) 2 ≥ d H ( N (0 , σ 21 ) , N (0 , σ 22 )) 2 .
Proof.
Using Lemma B.7, we have the following.
[Equation]
=
[Equation]
In the above, the second line follows from AM-GM inequality, that is, σ 21 + σ 22≥ σ 1 σ 2 .
This proves the first part.
Now, we prove the second part.
[Equation]
In the above, the second line follows from the fact that e − x ≤ 1 for x ≥ 0 .
This completes our proof.
The next lemma describes the relation between the parameters of two univariate Gaussians when their
TV distance is upper-bounded.
Lemma B.9 (TV Distance and Gaussian Parameters) .
Let G 1 ≡ N ( µ 1 , σ 21 ) , G 2 ≡ N ( µ 2 , σ 22 ) be
Gaussians over R .
Suppose σ max = max { σ 1 , σ 2 } and σ min = min { σ 1 , σ 2 } .
If d TV ( G 1 , G 2 ) ≤ α , then
[Equation]
Proof.
We have the following from Lemmata B.6 and B.8.
[Equation]
The last line holds because 1 1 − α = 1 + α 1 − α , and x ≥ ln(1 + x ) for all x ∈ R .
For the second part, we apply Lemmata B.6 and B.8 again to get the following.
[Equation]
Finally, we state a multivariate analogue of Lemma B.9.
Lemma B.10.
Suppose d TV ( N ( µ, Σ) , N ( (cid:101) µ, (cid:101) Σ)) ≤ γ for some γ < 1 .
Then
[Equation]
Proof.
Let G 1 ≡ N ( µ, Σ) and G 2 ≡ N ( (cid:101) µ, (cid:101) Σ) .
For any unit vector v ∈ R d , denote by v T G i the distribution over R obtained by sampling x ∼ G i and outputting v T x .
A data-processing inequality for the TV distance gives us that, for any unit vector v ∈ R d ,
[Equation]
where the first equality comes from the fact that the projection of a Gaussian is also Gaussian, with the above parameters.
By the second univariate bound in Lemma B.9, we have that for every unit vector v ∈ R d , (
[Equation]
− γ
[Equation] [Equation]
Rearranging the above gives us (1).
For (2), we have that for every unit vector v ∈ R d ,
[Equation]
where the last inequality comes from applying the first univariate bound in Lemma B.9.
Rearranging the above gives us (2).
B.2.2 Unknown Mean and Covariance with d + 1 Public Samples
Let L, U be quantities, as defined in Algorithm 1, and let
L γ =
(1 − γ ) 4· L and U γ = 4 (1 − γ ) 4 · U for 0 ≤ γ < 1 .
The following is an analogue of Lemma 2.5 for the case, where we apply Algorithm(public data preconditioning) with public data that comes from a Gaussian that is at most γ -far in TV distance from the private data distribution.
Lemma B.11 ( γ -Far Public Data Preconditioning) .
For all β > 0 , there exists an algorithm that takes d + 1 independent samples (cid:101)
X
=
( (cid:101) X 1 , . . .
, (cid:101)
X d +1 ) from a Gaussian N ( (cid:101) µ, (cid:101) Σ) over R d , and outputs (cid:98) µ ∈ R d , (cid:98) Σ ∈ R d × d , L γ ∈ R , and U γ ∈ R ,
such that for a Gaussian N ( µ, Σ) over R d with 0 ≤ d TV ( N ( µ, Σ) , N ( (cid:101) µ, (cid:101) Σ)) ≤ γ < 1 , if Σ Y = 1 L γ (cid:98) Σ − 1 / 2 Σ (cid:98) Σ − 1 / 2 and µ Y = 1 √ L γ (cid:98) Σ − 1 / 2 ( µ − (cid:98) µ ) , then with probability at least 1 − β over the sampling of the data,
I ⪯ Σ Y ⪯ U γ L γ
I 2.
∥ µ Y ∥ 2 ≤ (cid:113) U γ L γ · (cid:16)(cid:113) 10 γ 1 − γ + (cid:112) 5 log(3 /β ) (cid:17)
where U γ L γ =
O (cid:16) d 2 log(1 /β )
β 2 (1 − γ ) 8 (cid:17) .
Proof.
We prove the lemma by proving the utility of a modified version of Algorithm 1, which returns
L γ and U γ , instead of L and U , respectively.
The result follows from tracing through the proof of
Lemma 2.5, and applying Lemma B.10 as necessary.
We highlight the differences.
We start with (1).
By the same chain of equivalences in the proof of Lemma 2.5, it suffices to show that L γ Σ − 1 / 2 (cid:98) ΣΣ − 1 / 2 ⪯ I ⪯ U γ Σ
[Equation]
/ 2 (cid:98) ΣΣ − 1 / 2 .
We have the following.
[Equation] [Equation]
In the above, Z i , (cid:98) µ Z , and (cid:98) Σ Z are quantities, as defined in the proof of Lemma 2.5.
From the same proof, we know that with probability ≥ 1 − β/ 3 , we have U (cid:98)
Σ
Z ⪰ I , which implies that
[Equation]
where the last inequality follows from (1) in Lemma B.10.
Recalling that we set U γ = 4 (1 − γ ) 4 · U , and rearranging gives us that U γ Σ − 1 / 2 (cid:98) ΣΣ − 1 / 2 ⪰ I , as desired.
Similarly, with probability ≥ 1 − β/ 3 ,
L (cid:98) Σ Z ⪯ I , which implies that
[Equation]
which again, after rearranging, allows us to conclude L γ Σ
[Equation]
⪯ I .
It remains to verify that (2) holds.
Write µ
[Equation]
We bound the two terms separately.
Note that the second term appears in the proof of Lemma 2.5.
By the same argument as in that proof, we claim that with probability ≥ 1 − β 3 , ∥ (cid:98) µ Z ∥ 2 ≤ (cid:114)
[Equation]
d
[Equation]
We already know that (cid:13)(cid:13)(cid:13)(cid:98) Σ − 1 / 2 (cid:101) Σ 1 / 2 (cid:13)(cid:13)(cid:13) 2 ≤ √ U .
Taking the union bound, with probability ≥ 1 − β ,
[Equation]
Now, we argue that the first term is also bounded.
First, we apply Lemma B.10 to get
[Equation]
Σ .
Note that U (cid:98) Σ
Z ⪰ I =
⇒ (cid:101) Σ ⪯ U (cid:98) Σ .
Plugging this in above, and rearranging gives
[Equation]
Thus, we have
[Equation]
Combining the two terms gives us
[Equation]
which completes the proof.
Lemma B.11, combined with guarantees of the Gaussian estimators from Lemma A.24, allows us to conclude the following analogue to Theorem B.5 (this stated in the main body as Theorem 2.4).
Theorem B.12 ( γ -Far Public-Private Gaussian Estimation) .
For all α, β, ε, ρ > 0 , there exist ε -DP
and ρ -zCDP algorithms that take d + 1 public samples (cid:101)
X
=
( (cid:101) X 1 , . . .
, (cid:101)
X d +1 ) and n private samples
X =
( X 1 , . . .
, X n ) from unknown, d -dimensional Gaussians N ( (cid:101) µ, (cid:101) Σ) and N ( µ, Σ) , respectively,
such that 0 ≤ d TV ( N ( µ, Σ) , N ( (cid:101) µ, (cid:101) Σ)) ≤ γ < 1 , and are private with respect to the private samples, and return (cid:98) µ X ∈ R d and (cid:98)
Σ X ∈ R d × d , such that d TV ( N ( (cid:98) µ X , (cid:98) Σ X ) , N ( µ, Σ)) ≤
α with probability at least 1 − β , as long as the following bounds on n hold.
For ρ -zCDP (with computational efficiency) via [KLSU19] and d + 1 public samples,
[Equation]
For ε -DP (without computational efficiency) via [BKSW19] and d + 1 public samples,
[Equation]
Proof.
The theorem follows from the privacy and the utility guarantees of a modified version of
Algorithm 2, which uses the modified version of Algorithm 1 as outlined in Lemma B.11, and parameters L γ and U γ , instead.
The proof remains the same as that of Theorem B.5.
C Estimating Gaussian Mixtures
In this section, we state our algorithms to learn mixtures of well-separated Gaussians under differential privacy, when we have trace amounts of public data available to us.
We then provide theoretical guarantees for privacy of our algorithms with respect to the private data samples, along with its utility.
We analyse two cases here: when we have (cid:101) O
(1 /w min ) public samples available to us, and when we have (cid:101)
O ( d/w min ) public samples available.
The general scheme of our algorithms is similar to that of the non-private algorithm for learning
Mixtures of Gaussians by [AM05], and the private algorithm (with no public data available) by [KSSU19].
Many algorithms for learning mixtures of Gaussians follow a specific outline – use PCA to project the data onto a low-dimensional subspace, which would separate the largest Gaussian from the rest; partition the dataset again, if there is more than one Gaussian present, otherwise isolate the lone Gaussian; estimate the parameters of that Gaussian; repeat the process on the remaining points.
Our algorithms are also spectral algorithms that rely upon techniques, like PCA, but use their private counterparts at various stages.
The difference in the aforementioned cases, where we have O (1 /w min ) and O ( d/w min ) public samples available, is that in the former, we have very few public samples, but they are enough to be able to isolate a group of nearby clusters (called, a “supercluster”), which we could then use for an application of private PCA.
In the latter case, we wouldn’t need to apply private PCA at all because the number of public samples is enough to be able to do non-private PCA accurately.
Assumption.
We make an assumption about the shape of the covariances of all Gaussians, which essentially says that the Gaussians are not too flat or degenerate.
Let N be the total number of points sampled from D ∈ G ( d, k, s ) .
We formalise this as follows.
[Equation]
Note that this implies that d ≥ 8 log 2 ( Nk/β ) because tr(Σ i ) ≤ d ∥ Σ i ∥ 2 .
We also assume that β < 1 / 2 .
Notations.
We also define a few notations before moving on to the technical sections.
We say that a Gaussian N ( µ, Σ) in high dimensions satisfying Condition 2 is contained within S ⊂ R d , if
B √ 32 tr(Σ) ( µ ) ⊆ S .
For a low-dimensional Gaussian N ( µ, Σ ′ ) in ℓ < d dimensions, we say that a set
S ⊂ R d contains the Gaussian for fixed 0 < β < 1 and N ≥ 1 , if B √ ∥ Σ ′ ∥ 2 √ 2 ℓ ln(2 Nℓ/β ) ( µ ′ )
⊂ S .
Also, given D ∈ G ( d, k, s ) , we say that B ⊂ R d is “pure”, if for each Gaussian component i , B either contains the Gaussian N ( µ i , Σ i ) , or B √ 32 tr(Σ i ) ( µ i ) ∩ B = ∅ .
Similarly, given a set of points
T from D , we say that S ⊆ T is “clean”, if for every i ∈ [ k ] , S has all points from component i that lie in T , or it has none of them.
We sometimes say that for a clean S 1 ⊆ T , S 2 is a clean subset of Sif S 2 ⊆ S 1 and S 2 is clean with respect to T .
C.1 Deterministic Regularity Conditions
Here, we state a few results that would be useful in solving the problems in the next two subsections.
The following condition bounds the number of points from each component of a Gaussian mixture.
Condition C.1 (Sample Frequency) .
Suppose we have n samples from a mixture of Gaussians
D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( d, k, s ) satisfying Condition 2.
Then for each i ∈ [ k ] , the number of samples from component i lies in (cid:2) nw i 2 , 3 nw i 2 (cid:3) .
Furthermore, if w i ≥ 4 α 9 k , then the number of points from component i lies in (cid:2) n ( w i − α 9 k ) , n ( w i + α 9 k ) (cid:3) .
The next condition bounds the distance of any point sampled from a Gaussian mixture from the mean of its respective component.
Condition C.2 (Intra-Gaussian Distances From Mean) .
Suppose we have n samples from a mixture of Gaussians D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( d, k, s ) satisfying Condition 2.
Then for each i ∈ [ k ] , if x is one of the samples from component i , then 34 tr(Σ i ) ≤ ∥ x − µ i ∥ 2 ≤ 32 tr(Σ i ) .
This condition bounds the distance between any two points sampled from the same component of a
Gaussian mixture.
Condition C.3 (Intra-Gaussian Distances Between Points) .
Suppose we have n samples from
D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( d, k, s ) satisfying Condition 2.
Then for each i ∈ [ k ] , if x, y are two of the samples from component i , then 32 tr(Σ i ) ≤
∥ x − y ∥ 2 ≤ 3tr(Σ i ) .
This final condition quantifies the minimum distance between any two points from different components of a Gaussian mixture.
Condition C.4 (Inter-Gaussian Distances) .
Suppose we have n samples from a mixture of
Gaussians D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( d, k, s ) satisfying Condition 2, where s ≥ Ω( (cid:112) ln( n/β )) .
Then for any i, j ∈ [ k ] with i ̸
=
j , if x is one of the samples from component i , and y is one of the samples from component j , then ∥ x − y ∥ ≥ √ max { tr(Σ i )
[Equation]
tr(Σ j ) }.
We state another condition that says that in low dimensions, too, the distance between the mean of a
Gaussian from all its points is bounded, and it is true for all the components of the mixture.
Condition C.5 (Intra-Gaussian Distances from Mean in Low Dimensions) .
Suppose we have n samples from D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( ℓ, k ) .
For each i ∈ [ k ] , let σ 2 i = ∥
Σ i ∥ 2 .
Then for any i ∈ [ k ] and a fixed 0 < β < 1 , if x is a datapoint sampled from N ( µ i , Σ i ) , then ∥ µ i − x ∥ ≤ σ i (cid:112) 2 ℓ ln(2 nℓ/β ) .
The final condition about low-dimensional Gaussian mixtures bounds the distance between two points from the same component.
Condition C.6 (Intra-Gaussian Distances Between Points in Low Dimensions) .
Suppose we have n samples from D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( ℓ, k ) .
For each i ∈ [ k ] , let σ 2 i = ∥
Σ i ∥ 2 .
Then for any i ∈ [ k ] and a fixed 0 < β < 1 , if x, y are a datapoints sampled from N ( µ i , Σ i ) , then ∥ x − y ∥ ≤ 2 σ i (cid:112) 2 ℓ ln(2 nℓ/β ) .
Now, we prove that the above conditions hold with high probability.
Lemma C.7.
Suppose we have n samples from D ∈ G ( d, k, s ) satisfying Condition 2.
If n ≥ max
[Equation]
then Condition C.1 holds with probability at least 1 − β .
Proof.
We simply use Lemma A.1 k times with p ≥ w min , and Lemma A.2 k times, and take the union bound.
Lemma C.8.
Suppose we have n samples from D ∈ G ( d, k, s ) satisfying Condition 2.
Then
Condition C.2 holds with probability at least 1 − β .
Proof.
From the Hanson-Wright inequality (Lemma A.3), ∀ i, ∀ x ∼ N ( µ i , Σ i ) , we have that tr(Σ i ) −
[Equation]
i
[Equation]
Condition 2, combined with the above, gives us the required result.
Lemma C.9.
Suppose we have n samples from D ∈ G ( d, k, s ) satisfying Condition 2.
Then
Condition C.3 holds with probability at least 1 − β .
Proof.
For every i and for any x, y ∼ N ( µ i , Σ i ) , we have that x − y ∼ N (0 , 2Σ i ) .
Using Conditionand Lemma A.3 again, we have the result.
The following is a quantification of the median radius of a Gaussian in terms of the trace of its covariance.
Lemma C.10.
Suppose we have n samples from D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( d, k, s ) satisfying Condition 2.
Then for each i ∈ [ k ] , the median radius R i of component i lies in (cid:16)(cid:113)
tr(
[Equation]
) , (cid:113)
[Equation]
tr(Σ i ) (cid:17)
Proof.
For each i ∈ [ k ] , let a i = (cid:113)
tr(Σ i ) and b i = (cid:113) 32 tr(Σ i ) .
We know from the proof of
Lemma C.8 that for a given i ∈ [ k ] , 1 − βk of the probability mass of G i := N ( µ i , Σ i ) lies in
B b i ( µ i ) \ B a i ( µ i ) .
Let β 1 ≤ β/k be the probability mass of G i in B a i ( µ i ) , and let β 2 = β/k − β 1 .
We know that β 1 < 1 / 2 , therefore, R i > a i .
Since the mass of G i outside B b i ( µ i ) is β 2 < 1 / 2 , it must be the case that R
i <
b i .
Hence, the claim.
Lemma C.11.
Suppose we have n samples from D ∈ G ( d, k, s ) satisfying Condition 2.
If s ≥ Ω( (cid:112) ln( n/β )) , then Condition C.4 holds with probability at least 1 − 2 β .
Proof.
For this, we will use Lemmata A.12 and C.10.
We know from Lemma A.12 that with probability at least 1 − β/m , for any given i ∈ [ k ] , and x sampled from G i := N ( µ i , Σ i ) in the dataset sampled from D ,
[Equation]
For any j ∈ [ k ] with j ̸ = i , let z = µ j .
Suppose R i and R j are the median radii of G i and G j , respectively.
WLOG, let’s assume that R i ≥ R j .
Then we have the following.
[Equation]
(Separation condition and Condition 2.)
Now, let y ∼ G j .
Due to our separation condition, we know that,
[Equation]
We apply Lemma A.12 again with z = x
[Equation]
[Equation] [Equation]
(Condition 2.) ≥ tr
[Equation]
This proves the lemma.
Lemma C.12.
Suppose we have n samples from D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( ℓ, k ) .
Then for a fixed 0 < β < 1 , Condition C.5 holds with probability at least 1 − β .
Proof.
Fix i ∈
[ k ] , x be an arbitrary sample from component i , and an eigenvector of Σ i v .
In that direction, using Lemma A.4, we know that with probability at least 1 − βnℓ , ∥ ( x − µ i ) vv T ∥ ≤ σ i (cid:112) 2 ln(2 nℓ/β ) .
Applying the union bound in all ℓ directions of Σ i , we have that with probability at least 1 − βn , ∥ x − µ i ∥ ≤ σ i (cid:112) 2 ℓ ln(2 nℓ/β ) .
Applying the union bound again over all n points, we get the required result.
Lemma C.13.
Suppose we have n samples from D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } ∈ G ( ℓ, k ) .
Then for a fixed 0 < β < 1 , Condition C.6 holds with probability at least 1 − β .
Proof.
The proof just involves application of triangle inequality for any pairs of points from the same component, along with Lemma C.12.
C. (cid:101) O (1 /w min ) Public Samples
In this subsection, we assume that we have (cid:101) O
(1 /w min ) public samples available.
Here, the number of public samples is not enough to be able to do PCA accurately, so we, instead, try to use them to isolate the superclusters in the data, and perform private actions, like private PCA and private estimation algorithms, on the private data based on that information.
We start with a superclustering algorithm for public data.
It first computes a constant-factor approximation to the diamater ( ≈ r ) of the largest Gaussian in the public dataset by finding the largest minimum pairwise distance among all points.
It then tries to find a supercluster around the point ( x ) that has the largest minimum distance to all other points.
The idea is that B r ( x ) will have a lot of points, but if B r + r ( x ) doesn’t have any more points, and B r +2 r ( x ) doesn’t have any more points either, then it must be the case that with high probability, no other Gaussian would have a point in B 2 r ( x ) .
This follows from the concentration properties of high-dimensional Gaussians, and the fact that r approximates the diameter of the largest Gaussian.
If we do encounter more points in either B 2 r ( x ) or B 3 r ( x ) , we expand the ball by O ( r ) and check for these conditions again.
We show that the ball returned by the algorithm is of size at most O ( kr ) , and that it contains the Gaussian corresponding to the point x .
We now state the main result about Algorithm 3.
Theorem C.14.
There exists an algorithm, which if given a clean subset of
[Equation]
points from D ∈ G ( d, k ) , where D satisfies Condition 2, it outputs c ∈ R d and R ∈ R , such that the following holds, given Conditions C.1 to C.4 hold.
B R ( c ) is pure.
Let Σ ′ be the covariance of the largest Gaussian (covariance with maximum trace) contained in B R ( c ) .
Then R ∈ O ( k (cid:112) tr(Σ ′ )) .
Proof.
Let N ( µ, Σ) be
the
largest
Gaussian that has points in (cid:101)
X .
First, we claim that r ∈ Θ( (cid:112) tr(Σ)) .
We know from Lemma C.11, that the minimum distance between x ∼ N ( µ i , Σ i ) and y ∼ N ( µ j , Σ j )
Input: Samples (cid:101) X 1 , . . .
, (cid:101) X m ′ ∈ R d .
Parameters β, k, m > 0 .
Output: Centre c ∈ R d and radius R ∈ R .
Let r ← 16 · max x ∈ (cid:101)
X min y ∈ (cid:101) X
∥ x − y ∥ .
Let c ← arg max x ∈ (cid:101)
X min y ∈ (cid:101) X
∥ x − y ∥ .
Set pure ← F ALSE .
Let R ← r .
For i ← 1 , . . .
, k
Let m i ← (cid:12)(cid:12)(cid:12) B R ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12) .
If (cid:12)(cid:12)(cid:12) B R + r ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12) =
m i
If pure = T RUE
Return B R ( c ) elif (cid:12)(cid:12)(cid:12) B R
+2 r ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12) =
m i
Return ( c, R + r )
Else pure = F ALSE
R ← R + 3 r Else
If (cid:12)(cid:12)(cid:12) B R +2 r ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) B R + r ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12) pure = T RUE
Else pure = F ALSE
R ← R + 2 r
Return ( c, R ) .
for i ̸
= j is lower bounded by √ max { tr(Σ i ) , tr(Σ j ) }.
We also know from Lemma C.9 that for any i ∈ [ k ] , and any x, z ∼ N ( µ i , Σ i ) , ∥ x − z ∥ ≥ (cid:113) 32 tr(Σ i ) .
Let y ∈ (cid:101) X , such that y is sampled from component j ̸
= i and is closest to x , that is, y is the closest point to x that has not been sampled from component i to which x belongs.
This means that distance between x and its nearest point is at least min (cid:26)(cid:113) 32 tr(Σ i ) ,
[Equation]
We also know from Lemma C.9 that for any x, z ∼ N ( µ i , Σ i ) , ∥ x − z ∥ ≤ (cid:112) 3tr(Σ i ) .
This means that the point in (cid:101) X that is closest to x is at most (cid:112) 3tr(Σ i ) far from x .
This shows that the distance between x and the point in (cid:101) X that is closest to x lies in (cid:20) √ tr(Σ i )
[Equation]
This is true for any point in (cid:101) X .
Therefore, the largest minimum distance has to lie in the interval
[Equation]
Σ
[Equation]
This shows that r ∈ Θ( (cid:112) tr(Σ)) .
We use this to show that B R ( c ) is pure.
We prove the following claims for that.
Claim C.15.
If at the end of any iteration, the algorithm doesn’t exit the loop, but calls any of the return steps within the loop instead, then the returned ball is pure.
Proof.
Suppose it happens in iteration i .
Consider the case where the return statement was called in the elif block.
If B R ( c ) were intersecting with a component, then B R + r ( c ) would contain it (Lemmata C.8 and C.9) because r is large enough.
Now, we need to show that B R + r ( c ) doesn’t intersect with any new component.
By construction, it must mean that (cid:12)(cid:12)(cid:12) B R ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12)
=
m i , but
(cid:12)(cid:12)(cid:12) B R + r ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12) =
m i and (cid:12)(cid:12)(cid:12) B R +2 r ( c ) ∩ (cid:101) X (cid:12)(cid:12)(cid:12)
= m i .
If B R + r ( c ) intersects with another component that it doesn’t fully contain, then it must be the case that the component would have at least one more point in B R +
r ( c ) \ B R + r ( c ) and would be contained in that region (by Lemma C.8) because r is large enough.
Since that doesn’t happen, there cannot be any component that is partially contained in
B R + r ( c ) .
Therefore, B R + r ( c ) is pure.
If return were called because pure were true, then it must have been the case that in iteration i − 1 , before the final update to R , pure must have been set to T RUE because B R + r ( c ) found new points, but B R +2 r ( c ) couldn’t.
In all other case, pure is set to F ALSE .
Since in iteration i , B R + r ( c ) couldn’t find new points either, by the same reasoning as above, B R ( c ) must be pure.
Claim C.16.
In iteration i , either the algorithm exits with a pure ball containing at least i −components, or at the end of iteration i , B R ( c ) contains at least i components.
Proof.
We show this by induction on i , as defined in Algorithm 3.
Base Case: At the beginning of the first iteration, R = r .
But we know from Lemmata C.9 and C. that r is big enough that the mean of the Gaussian from which c has been sampled, along with all other points from the same Gaussian would be contained in B r ( c ) because the mean of the Gaussian is at most (cid:113) 32 tr(Σ) away from c and all other points from the same component are at most (cid:112) 3tr(Σ) away from c .
Now, if the algorithm does not exit after the end of the iteration, our base case holds because we have at least one component in the ball at end of the iteration.
If the algorithm exited, it must be the case that B 2 r ( c ) is pure (from the above claim) and has at least one component.
Inductive Step: Suppose for i ≥ 1 , assume that at the end of iteration i , B R ( c ) contains at least
i
Gaussian components, or that it exits with at least i − 1 components.
If the algorithm exits in iteration i , we are done.
So, we assume that it doesn’t.
In iteration i + 1 , if the algorithm exits and returns a ball, then it is pure (by the claim above and the inductive hypothesis), and has at least i components.
So, we’re done.
Otherwise, either B R + r ( c ) or B R +2 r ( c ) finds new points.
If B R + r ( c ) finds new points, it is fully containing a component not previously contained within B R ( c ) , or it is intersecting with new components.
In the former case, a new component is added by the end of the iteration, and our claim holds.
This holds in the latter case, too, by Lemmata C.8 and C.9 because B R +2 r ( c ) would contain that new component.
So, we have contained at least i + 1 components at the end of iteration i
+ 1 .
If B R + r ( c ) has no new points, but B R +2 r ( c ) does, then these must be points from a new component.
Therefore, B R +3 r ( c ) would completely contain the new component by Lemmata C. and C.9.
So, we have i + 1 components again by the end of the iteration.
From the above argument, we have that either the algorithm exited in the loop, and returned a pure ball, or it had k components at the end of iteration k , in which case, the returned ball is again pure.
This gives us the first part of the theorem.
Now, we prove the second part.
Let c ∈ (cid:101)
X be as defined in Algorithm 3.
Let Σ be the covariance of the largest Gaussian G in (cid:101) X .
Suppose c was sampled from component G ′ :
= N ( µ ′ , Σ ′ ) .
If tr(Σ)
= tr(Σ ′ ) , then by the proof above, r ∈ Θ( (cid:112) tr(Σ)) .
Suppose tr(Σ ′ ) <
tr(Σ) .
Then in the worst case, the nearest point to c is (cid:112) 3tr(Σ ′ )
away from c .
It is possible in this case that all points from the largest Gaussian had nearest points √ tr(Σ) away from them, but (cid:112) 3tr(Σ ′ ) ≥ √ tr(Σ) .
In this case r would still be 16 √ 3 -factor approximation of (cid:112) tr(Σ ′ ) .
We know that the returned ball
B R ( c ) contains the component of c .
Also, the loop of the algorithm runs at most k times, where in each iteration, R can only increase additively by 3 r .
Therefore, the final radius can be at most (3 k + 1) r
∈
O
( k (cid:112) tr(Σ ′ ) .
This proves the second part of the theorem.
Now, we restate a folklore private algorithm for obtaining a projection matrix of the top- k subspace of a dataset via PCA, along with its main result.
Here, we just talk about its utility, but focus on the privacy later when we instantiate it for different forms of differential privacy.
Given a dataset, it simply truncates all points to within a given range, computes the empirical covariance matrix, then adds random noise to each entry of the matrix that is sampled from a Gaussian distribution scaled according to the appropriate privacy parameters depending on the notion of privacy.
In the algorithm, f PCA (PrivParams) denotes an appropriate function of the privacy parameters, which we would set according to the type of differential privacy guarantee we require.
Let f PCA be an appropriate function of the elements of PrivParams to ensure DP.
Set σ P ← 2 r 2 f PCA (PrivParams) .
Truncate all points of X to within B r (0) to get dataset Y .
Let E ∈ R d × d , such that for all i, j ∈ [ d ] with i ≤ j , E i,j ∼ N (0 , σ 2 P ) and E j,i ← E i,j .
Let Π be the projection matrix of the top- ℓ subspace of Z ← Y T Y + E .
Return
Theorem C.17 (Theorem 9 from [DTTZ14]) .
Let X, Y, Z, ℓ, c, r, σ P be quantities as defined in
Algorithm 4, C ℓ be the best rank- ℓ approximation to Y T Y , and (cid:98) C ℓ be the rank- ℓ approximation to Z .
Then with probability at least 1 − β ,
[Equation]
We get the following corollary using Theorem C.17 and Lemmata A.13 and A.14.
Corollary C.18.
Let X, Y, Z, ℓ, c, r, σ P , Π be quantities as defined in Algorithm 4, such that X is a clean subset of n samples from a mixture of k Gaussians D satisfying Condition 2, and X contains points from k ′ components.
Suppose w min is the minimum mixing weight of a components with respect to D .
Let σ 2max be the largest directional variance of any component that has points in X , and Σ be the covariance of the Gaussian with largest trace that has points in X .
If r ∈ O ( k (cid:112) tr(Σ)) , all points in X lie within B r (0) , and
[Equation]
then with probability at least 1 − O ( β ) , for each component ( µ ′ , Σ ′ , w ′ ) that has points in X ,
[Equation]
Proof.
By construction of our algorithm and our assumption on r , we know that
O
[Equation]
(since D satisfies Condition 2).
Using Lemma A.7 and our bound on n , we know that for each component ( µ ′ , Σ ′ , w ′ ) , the empirical mean of all the points from that Gaussian in X would be at most √ ∥ Σ ′ ∥
2 √ w ′ away from µ ′ .
By the same reasoning, the empirical mean of the same projected points would be at most √ ∥ Σ
[Equation]
√ w ′ away from µ ′ Π because the projection of a Gaussian is still a Gaussian.
Lemmata A.13 and A.14, and Theorem C.17 together guarantee that the distance between the empirical mean of those points and that of the projected points of that component would be at most O (cid:16) σ max √ w ′ (cid:17) .
Applying the triangle inequality gives us the result.
Now, we mention a result about the sensitivity of Y T Y in Algorithm 4, which would be used in later subsections.
Lemma C.19.
In Algorithm 4, the ℓ 2 sensitivity of Y T Y is 2 r 2 .
Proof.
Since all points in X are truncated to within B r (0) , for all i , ∥
X i ∥ ≤ r .
Therefore, for a neighbouring dataset X ′ , and corresponding truncated dataset Y ′ , ∥ Y T Y − Y ′ T Y ′
∥
F ≤ 2 r 2 .
Next, we describe an algorithm for partitioning the data after the PCA step has been performed.
For this, we assume that the data is in low-dimensions ( ℓ -dimensional), where Condition 2 may not hold, but there is at least one cluster that is well-separated from the other points, or there is just one cluster in the whole dataset.
We provide a private partitioner for private and public data for this regime, which in the first case, returns a partition of the datasets that contains their clean subsets, and in the second case, it returns the same cluster itself.
For this, we define the following queries (akin to those in [KSSU19]) that would be used for the public and the private data (respectively) in the algorithm.
[Equation]
Just like in [KSSU19], we define the notion of a terrific ball .
We say a ball B r ( c ) is ( γ, t ) - terrific with respect to some dataset X for γ > 1 , t > 0 , if (1) B r ( c ) contains at least t points from X , (2)
B γr ( c ) \ B r ( c ) contains at most t 80 points from X , and (3) R d \ B γr ( c ) contains at least t points from X .
We sometimes omit the parameter t when the context is clear.
For i ← 0 , . . .
, log( r max /r min ) r
i ← r max 2 i
For j ← 1 , . . .
, m ′ c
j ← (cid:101) Y j
If Q Pub ( (cid:101) Y , c j , r i , mw min)
=
T RUE
If Q Priv ( (cid:101) Z, c j , 2 r i , nw min, PrivParams)
=
T RUE
Return ( c = c j , R = 2 r i ) .
ElseReturn ⊥ .
Return ⊥ .
Now, we prove the main theorem about the utility of Algorithm 5.
In the following, f PCount (PrivParams) denotes an appropriate function of the privacy parameters based on the accuracy guarantees of PCount (Lemma A.23) depending on the notion of DP being used.
We prove the privacy guarantees in later sections.
Theorem C.20.
Let Y be a clean subset of a set of public samples from D ∈ G ( d, k ) , Z be a clean subset of private samples from D , Π be a projection matrix to ℓ dimensions, (cid:101)
Y =
Y Π , (cid:101)
Z
= Z Π be the input to Algorithm 5, r max , r min > 0 , such that for some j ∈ [ m ′ ] , (cid:101) Y ∩ B r max (cid:16)(cid:101) Y i (cid:17) =
(cid:101) Y and (cid:101)
Z ∩ B r max (cid:16)(cid:101) Y i (cid:17)
= (cid:101) Z and B r max (cid:16)(cid:101) Y
i (cid:17) is pure with respect to the components of D projected on to the subspace of Π having points in (cid:101) Y (and (cid:101) Z ).
Suppose σ 2max is the largest directional variance among the Gaussians that have points in Y .
Let that Gaussian be N ( µ, Σ) .
Suppose,
[Equation]
and
[Equation]
Then we have the following with probability at least 1 − 4 β .
Suppose there are at least two Gaussians that have points in Y , and the same Gaussians have points in Z .
For any other Gaussian N ( µ ′ , Σ ′ ) that has points in Y and Z , suppose for N = m + n , ∥ µ Π − µ ′ Π ∥ ≥ Ω( σ max (cid:112) ℓ ln( Nℓ/β )) .
Let r min ≤ σ max (cid:112) 2 ℓ ln(2 Nℓ/β ) .
Then the ball returned by Algorithm 5 is pure with respect to the components of D projected by Π , which have points in (cid:101) Y , such that R d \ B R ( c ) only contains components from D projected by Π that have points in (cid:101) Y (alternatively, (cid:101) Z ), and so does B R ( c ) .
Suppose there is only one Gaussian that has points in Y and Z .
Then the algorithm returns ⊥ .
Proof.
We first prove the first part of the theorem.
In this case, we know that there exists a ball of some radius r > 0 and some centre c ∈ (cid:101) Y , such that B r ( c ) has the required properties.
We prove the existence of such a ball first.
Let c ∈ (cid:101)
Y be a point, such that the corresponding Gaussian to c in d dimensions is N ( µ, Σ) (with ∥ Σ ∥ =
σ 2max ).
For any Gaussian N ( µ ′ , Σ ′ ) that has points in Y , in the subspace of Π , ∥ µ Π − µ ′ Π ∥ ≥ 103 σ max (cid:112) ℓ ln( Nℓ/β ) .
Now, we know that for any Gaussian
N ( µ ′′ , Σ ′′ ) , in this subspace, for any y ∼ N ( µ ′′ , Σ ′′ ) that lies in Y and any z ∼ N ( µ ′′ , Σ ′′ ) that lies in Z , ∥ y Π − µ ′′ Π ∥ , ∥ z Π − µ ′′ Π ∥ ≤ (cid:112) ∥ ΠΣ ′′ Π ∥ 2 (cid:112) 2 ℓ ln(2 Nℓ/β ) from Lemma C.12.
The triangle inequality implies that for any y, z ∼ N ( µ, Σ) with y ∈ Y and z ∈ Z , and a, b ∼ N ( µ ′ , Σ ′ ) with a ∈
Y and b ∈ Z , ∥ y Π − a Π ∥ , ∥ z Π − b Π ∥ ≥ 100 σ 2max (cid:112) ℓ ln(2 Nℓ/β ) .
Given that B 2 σ max √ 2ln(2 Nℓ/β ) ( c ) contains all points in Y Π and Z Π , whose Gaussian in high dimensions is N ( µ, Σ) , and contains the
Gaussian in low dimensions, this ball satisfies the required properties.
Now, let’s say the algorithm returns a ball B = B 2 r i ( c j ) .
We know that in this case, because the radius r i decreases when i increases, r i ≥ 2 σ max (cid:112) 2 ℓ ln(2 Nℓ/β ) .
Because r i is bigger than the largest possible diameter of any Gaussian component in the ℓ -dimensional subspace, B r i ( c j ) must be, at least, containing the component of c j in that subspace.
We know that the shell S =
B 11 r i ( c j ) \ B r i ( c j ) has no points from any component in (cid:101) Y (by construction).
For any component inside B r i ( c j ) (either partially or completely), the distance between the mean of any such component from the mean of any other component that is completely outside B r i ( c j ) has to be at least 10 r i − 2 × r i 2 =
9 r
i (because r i is at least twice the maximum distance from the mean of a Gaussian to any of its points in that subspace).
If there a component, whose mean lies in the shell T = B 21 ri 2 ( c j ) \ B 3 ri 2 ( c j ) , it would be completely contained inside S , and we would see points of (cid:101) Y in S , which would be a contradiction.
Therefore, the means of all components that lie completely outside B r i ( c j ) must be within R d \ B 21 ri 2 ( c j ) , which means that they would be entirely contained within R d \ B 10 r i ( c j ) (because of Lemma C.12).
Therefore, B cannot have any component that lies completely outside
B r i ( c j ) .
By similar reasoning, the mean of any component that partially intersects with B r i ( c j ) has to be within B 3 ri 2 ( c j ) .
Therefore, by Lemma C.12 again, that component would be contained within B 2 r i ( c j ) .
Therefore, the ball B is pure.
By the construction of the query Q Priv , our sample complexity (of n ), and the guarantees of PCount , with probability at least 1 − 3 β , we know that the noise in each call to PCount cannot be more than nw min 1280 (Lemma A.23).
We also know that the number of points from each component is at least nw min(from Lemma C.7).
Since B 2 r i ( c j ) is pure, it must have at least nw minpoints in it, so the noisy answer via PCount would be at least nw min.
Similarly, since there is no Gaussian component even partially contained within
B 10 r i ( c j ) \ B 2 r i ( c j ) , the noisy answer will be less than nw min 1280 .
Finally, since there is at least one component in R d \ B 10 r i ( c j ) , the noisy answer of PCount would be over nw min.
So, Q Priv would return T RUE .
This proves the first part of the theorem.
Now, we prove the second part of the theorem.
It is sufficient to show that there exists no 5 -terrific ball in the private dataset (cid:101)
Z when all the data is coming from a single Gaussian.
First, we claim that if there is a 5 -terrific ball in a dataset X , then there exists a 3 -terrific ball, too.
The argument is simple.
Suppose B r ( c ) is a 5 -terrific ball, then it is easy to see that B r ′ ( c ) for r ′
= 5 r 3 is a 3 -terrific ball.
This is because B r ′ ( c ) would contain more points that B r ( c ) , hence, B 3 r ′ ( c )
\ B r ′ ( c ) would contain fewer points than B 5 r ( c ) \ B r ( c ) .
Since R d \ B 5 r ( c ) =
R d \ B 3 r ′ ( c ) , the final constraint would hold, as well.
Hence, the claim.
So now, it is sufficient to show that there exists no 3 -terrific ball with respect to (cid:101) Z .
We prove this via contradiction.
We know that due to our sample complexity (bound on n ), the noise in each call to PCount is at most nw min 1280 .
Therefore, for a query Q Priv ( (cid:101) Z, c ′ , r ′ , nw min, PrivParams) to return a ball B r ′ ( c ′ ) , B r ′ ( c ′ ) and R d \ B 5 r ′ ( c ′ ) must contain at least nw minpoints from (cid:101) Z , and
B 5 r ′ ( c ′ )
\ B r ′ ( c ′ ) must contain at most nw min 640 points from (cid:101) Z .
This implies the existence of a 3 -terrific ball of radius r with respect to (cid:101) Z .
This means that there exists a unit vector v such that on projecting all the data on to v , we would have an three adjacent intervals along the vector
I 1 , I 2 , I 3 of length 2 r each along the direction of v , such that I
and I 3 together contain at most nw min 640 points, I 2 contains at least nw minpoints, and the rest of the line contains at least nw minpoints.
Suppose the mixing weight of the Gaussian in question is w i .
Lemma C.7 shows that the number of points in (cid:101) Z is Cnw i , where 12 ≤ C ≤ 32 .
A straightforward application of Chernoff bound (Lemma A.1) and the union bound over all 2 O ( d ) unit vectors in a cover of the unit sphere in R d , along with our sample complexity, implies that the probability mass contained in I 2 and the part of line excluding I 1 , I 2 , and I 3 (which we call, I 4 ) is at least w min 8 Cw i each, and the mass contained in I 1 ∪ I 3 is strictly less than w min 160 Cw i .
We show that this is impossible when the underlying distribution along v is a (one-dimensional)
Gaussian.
Suppose the variance along v of the said Gaussian is σ 2 .
WLOG, we will assume that its mean is 0 .
Note that for the said condition about the probability mass to be true, I 2 must contain the mean.
This is because of the symmetry about the mean and unimodality of one-dimensional Gaussians – if
I 2 does not contain the mean, then either I 1 or I 3 would be closer to the mean, and would contain more mass than I 2 because all three of these intervals are of the same length.
Next, we show that the total mass contained within I
and I 3 is minimised when the mean is at the centre of I 2 .
After that, it would be sufficient to show that in this configuration, the probability mass contained within Iand I 3 together is at least w min 160 Cw i when the mass contained in I
and I 4 each is at least w min 8 Cw i .
Let 0 ≤ r 1 ≤ r and r 2 = 2 r − r 1 , such that one end point of I 2 is r 1 away from the mean, and the other end point is r 2 away from the mean.
Then the probability mass contained within I 1 ∪ I 3 is
[Equation]
Define f ( r 1 ) = 1 √ 2 π
[Equation]
2 dt .
Then we have the following.
[Equation]
(cid:21)
Within the interval [0 , r ] , setting the above to 0 , we have r 1 = r .
It can be checked that for all r 1 ∈ [0 , r ] , d 2 f ( r 1 ) dr> 0 .
This means that within [0 , r ] , f is minimised at r 1 = r .
Therefore, for the rest of the proof, we would concentrate on lower bounding the quantity q ( r ) = 2 √ 2 π 3 r/σ (cid:82) r/σ e − t 2 2 dt .
We define p ( r ) = 2 √ 2 π r/σ (cid:82) e − t 2 2 dt to be the mass of I 2 in this regime, and λ ( r ) = 2 √ 2 π ∞ (cid:82) 3 r/σ e − t 2 2 dt to be the mass of I 4 .
We consider three cases.
Case 1: r 2 = σ 2 + τ , 0 ≤ τ ≤ σ 2 .
Noting that e − x +0 .
25 ≥ e − x 2 for x ∈ [0 , 1] and 1 √ 2 ≤ r √ 2 σ ≤ 1 , we have the following.
[Equation]
Now, we compute q ( r ) .
[Equation]
=
[Table]
 
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We have a contradiction.
Thus, in this case, the probability mass in I 1 ∪ I 3 has to be at least w min 160 Cw i .
Case 2: r 2 =
σ 2 + τ , τ ≥ σ 2 .
We have the following upper bound on λ ( r ) using Lemma A.4.
[Equation]
We know that λ ( r ) ≥ w min 8 Cw i .
This implies that r 2 σ 2 ≤ 29 ln (cid:16) 16 Cw i w min (cid:17) .
This is also equivalent to w min 16 Cw i ≤ 1 e 9 .
Lemma A.6 gives us the following lower bound on 1 − p ( r ) .
[Equation]
This gives us the following lower bound on q ( r ) .
[Equation]
This gives us a contradiction for the second case, as well.
Case 3: r 2 = σ 2 − τ , 0 ≤ τ < σ 2 .
This final case has two sub-cases to analyse: 9 r 2 ≤ 2 σ 2 and 9 r 2 > 2 σ 2 .
In the first instance, we have the following bounds on p ( r ) .
[Equation]
Next, we lower bound p (3 r ) .
[Equation]
This gives us the following lower bound on q ( r ) .
[Equation]
This shows that this situation is impossible.
So, we move on to the final instance, that is, when 2 σ 2 9 < r 2 < σ 2 .
Just as we did in Case 1, we bound p ( r ) .
We get the following.
[Equation]
We lower bound p (3 r ) again.
[Equation]
Now, we lower bound q ( r ) .
[Equation]
[Figure]
This gives us the final contradiction, hence, completing the proof for Case 3.
Therefore, there cannot be such intervals on the line of v .
This means that there cannot be any such direction, where such intervals exist, which means that there cannot be a 3 -terrific ball with respect to (cid:101) Z , implying that the algorithm could not have found a 5 -terrific ball with respect to (cid:101) Z .
By the construction of the algorithm, the output would be ⊥ .
This completes the proof of the theorem.
We now state the main private algorithm for clustering points from a Gaussian mixture (Algorithm 6) that utilises the public data available to it, along with a theorem describing its utility guarantees.
In the theorem, f ′ PCA and f ′ PCount denote appropriate functions of the privacy parameters in terms of the previously mentioned functions f PCA and f PCount , respectively.
Note that the output of the algorithm itself is not private, but the intermediate steps involved are.
Since its output will not be released to public, and will just be utilised by our main private estimation algorithm, there will be no violation of privacy in the end.
This algorithm is a general private framework, which could be instantiated for approximate DP and zCDP by choosing the right parameters and privacy primitives, for example, it would use approximate DP PCA algorithm for the former, and zCDP
PCA algorithm for the latter.
It first uses Algorithm 3 on the public data to find a supercluster that tightly bounds a group of clusters.
Since the private data is sampled from the same distribution as the public data, it uses that supercluster to isolate the points in the private dataset corresponding to the components contained within the supercluster.
Next, it uses private PCA to project the data on to the top- k subspace, and then works within that subspace to either determine if the data is coming from a single Gaussian, or to partition the dataset into clean subsets in case it is coming from at least two
Gaussians.
In the former case, it adds the set of private points in the original d -dimensional space to the output (which contains sets of points).
In the latter case, it just proceeds to work on the partitions independently.
Theorem C.21.
There exists an algorithm (Algorithm 6) that takes n private samples X , and m public samples from D ∈ G ( d, k, s ) satisfying Condition 2, and outputs a partition C of X , such that given Conditions C.1 to C.6 hold, and
[Equation]
and
[Equation]
, then with probability at least 1 − O ( β ) , | C | =
k and each S ∈ C is clean and non-empty.
Proof.
It is enough to show that ∀ i ≥ 0 , at the end of the i -th iteration, (1) a new Gaussian component is isolated in C and count equals the number of isolated components in C , (2) or the private dataset
X is further partitioned into non-empty, clean subsets, and the public dataset (cid:101) X is further partitioned into clean subsets, such that | Q Pub | = | Q Priv | and for each j ∈ [ | Q Pub | ] , Q Pub [ j ] and Q Priv [ j ] have points from the same Gaussian components.
We can prove this via induction on i .
Base Case: The end of the 0 -th iteration essentially means the actual start of the loop.
In this case, we know that Q Pub and Q Priv are clean subsets of (cid:101) X and X , respectively.
Therefore, the claim trivially holds in this case.
Inductive Step: We assume for all iterations, up to and including some i ≥ 0 , the claim holds.
Then we show that it holds for iteration i + 1 , as well.
By the inductive hypothesis, we know that Z and Y are clean subsets, and contain points from the same components.
When the superclustering algorithm (Algorithm 3) is called, it finds a pure ball that will either contain the whole dataset Y (hence, Z because the ball is pure), or it partitions Y into clean subsets (Theorem C.14).
Since the components from which the points in Y and Z come from are the same,
B R i
( c i ) would contain points from the same components from Z as in Y (because the deterministic regularity conditions hold).
So, adding
Z \ Z i to Q Priv and Y \ Y i to Q Pub ensures that the partition of Z added to the back of Q Priv is clean and contains the points from the same components as those that have points in Y \
Y i (which itself gets added to the back of Q Pub ), which preserves the ordering of the set of components in the two queues.
If there is just one component that has points in Y i (hence, in Z i ), then after projecting the data points on to the subspace of Π i , and feeding the projected datasets to Algorithm 5, we will get ⊥ (by the guarantees from Theorem C.20) with probability at least 1 − β 2 k .
Because Z i is a clean subset, C now contains one more clean subset of X , and we have isolated a new component, and updated the value of count to reflect the change.
Input: Private samples X
=
( X 1 , . . .
X n ) ∈ R n × d .
Public samples (cid:101) X
=
( (cid:101) X 1 , . . .
, (cid:101)
X m ) ∈ R m × d .
Parameters PrivParams ⊂ R , β, k, w min > 0 .
Output: Set C ⊂ P ( X ) .
Let Y ← (cid:101)
X and Z ← X .
Let C ← ∅ .
Let Q Priv and Q Pub be queues of sets of points.
Add set Z to Q Priv , and set Y to Q Pub .
PrivParams ′ be the set of privacy parameters modified as per composition based on
PrivParams .
Set count ← 0 and i ← 1 .
While count < k and i ≤ 2 k
Pop Q Priv to get Z , and Q Pub to get Y .
// Run superclustering algorithm on the public dataset.
Set ( c i , R i )
← SC k,m ( Y ) .
// Partition both datasets on the basis of the supercluster.
Y i ← Y ∩ B R
i ( c i ) and Z i ← Z ∩ B R i ( c i ) .
Add Z \ Z i to Q Priv , and Y \ Y i to Q Pub .
Private PCA: project points of both datasets onto returned subspace.
Let M i ∈ R | Z i |× d , such that each row M ij ← c
i .
Π i ← PrivPCA PrivParams ′ ,k
( Z i − M i , R i ) .
Y ′
← ( Y i − M i )
Π i and Z ′ ← ( Z i − M i )Π
i .
// If there is only one Gaussian, add it to C , otherwise further partition the datasets.
B ← DPLowDimPartitioner n,m,w min , PrivParams ′
( Z ′ , Y ′ ,
R i , R i √ d ) .
If B = ⊥
C ← C ∪ {
Z i } .
count ← count + 1 .
Else B is an ordered pair ( c ′ i , r ′ i ) .
S ′ ← B
r
i ( c ′ )
∩ Y ′ and T
′ ← B r
i ( c ′ )
∩ Z ′ .
Let S be points in Y i corresponding to S ′ , and T be points in Z i corresponding to T ′ .
Add S to Q Pub and T to Q Priv .
Add Y i \ S to Q Pub and Z i \ T to Q Priv .
i ← i +Return
Suppose there are at least two components that have points in Y
i (hence, in Z i ).
Then we know from
Theorem C.14, that if Σ is the covariance having the largest trace among all components that have points in Y i (hence, in Z i ), then R i
∈
O ( k (cid:112) tr(Σ)) .
Let the largest directional variance among the
Gaussians that have points in Z i (hence, in Y i ) be σ 2max , its mean be µ , and its mixing weight be w .
From the guarantees in Corollary C.18, we know that ∥ µ − µ Π i ∥ ≤ O (cid:16) σ max √ w (cid:17) with probability at least 1 − β 2 k .
Because of the separation condition, applying the triangle inequality, we know that the distance between µ Π i and the projected mean of any other Gaussian having points in Z i Π is at least Ω( σ max (cid:112) k
ln(( n + m ) k ) /β ) .
Then from the guarantees of Theorem C.20, we have that a ball is returned, and it is pure with respect to the components of the mixture projected by Π i that have points in Z i Π .
Therefore, subsets S ′ and T ′ are clean, and so are S and T .
This implies that Y i \ S and Z i \ T are clean, too.
By adding them in order to Q Pub and Q Priv respectively, we partition the datasets (cid:101) X and X further into clean subsets, and preserve the ordering of the components in the respective queues.
This proves the claim.
Note that before the PCA step, we recentre the points in both Y i and Z i for the ease of analysis later when we provide guarantees for privacy.
It does not affect correctness of any step of the algorithm because it is just recentering of data.
Now, we just have to show that the number of iterations we allow is enough with high probability.
Because we have k components in the mixture, we can only partition into clean subsets at most k −times.
For each component, the loop will run at most one time.
So, 2 k − 1 iterations are enough to capture all the components, and | C | =
k .
Taking the union bound over all possible 2 k − 1 iterations, we have the required result.
We finally state the private algorithm (Algorithm 7) for estimating the parameters of mixtures of
Gaussians, along with a theorem highlighting its utility guarantees.
Just like the DP clustering algorithm above, it is a private framework that could instantiated for approximate DP and zCDP settings by using the corresponding private algorithms.
It calls Algorithm 6 to cluster the points according to their respective components, then calls private estimators to estimate the parameters of the respective components.
It finally uses PCount to estimate their mixing weights based on the private data.
Input
Private
Public samples (cid:101) X
=
( (cid:101) X 1 , . . .
, (cid:101)
X m ) ∈ R m × d .
Parameters PrivParams ⊂ R , β, k, w min > 0 .
Output
// Clustering.
Set C ← DPHardClustering PrivParams ,β,k,w min ( X, (cid:101) X ) .
Parameter estimation.
(cid:98) D ← ∅ .
For i ← 1 , . . .
, k
Set ( (cid:98) µ i , (cid:98) Σ i ) ← DPGaussianEstimator PrivParams , βk ( C i ) .
Set (cid:98) w i ← max (cid:8) 1 n · PCount PrivParams ( | C i | ) , α 2 k (cid:9) .
(cid:98) D ← (cid:98) D ∪ { ( (cid:98) µ i , (cid:98) Σ
i , (cid:98) w i ) } .
Return (cid:98) D .
Theorem C.22.
For all α, β > 0 and sets of privacy parameters PrivParams , there exists an algorithm (Algorithm 7) that takes n private samples and m public samples from D ∈ G ( d, k, s ) , such that if D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } satisfies Condition 2, and s ∈ Ω( (cid:112)
[Equation]
n +
[Equation] [Equation]
n ∈
O
[Equation]
(cid:33) m
∈ O (cid:18) ln
( k/β ) w min (cid:19) , where n GE is the sample complexity of privately learning a Gaussian using Lemma A.24 according to the type of DP required, then it ( α, β ) -learns D .
Proof.
By our sample complexity bound and Theorem C.21, with probability at least 1 − β 3 , for each i ∈ [ k ] , C i is clean with respect to X .
Therefore, each C i can be used to learn an independent component.
Next, by the guarantees of Lemma A.24, for each i ∈ [ k ] , d TV ( N ( (cid:98) µ i , (cid:98) Σ i ) , N ( µ i , Σ i )) ≤ α with probability at least 1 − β 3 k .
Finally, using Lemmata C.7 and A.23, along with our sample complexity, we know that the error due to each call to PCount is at most α 2 k with probability at least 1 − β 3 k , and that the error due to sampling is at most α 2 k .
We apply the triangle inequality to get the final error of αk for each mixing weight.
Applying the union bound over all failure events (including the failures of Conditions C.1 to C.6), we have the desired result.
C.2.1 ( ε, δ ) -DP Algorithm
Here, we describe our results under approximate DP constraints, and instantiate Algorithms 4, 6, andfor this version of DP.
We will not restate the entire algorithms, but just describe how PrivParams and the sample complexity in these different algorithms change.
We now state the main theorem of our section.
Theorem C.23.
For all α, β, ε, δ > 0 , there exists an ( O ( ε ) , O ( δ ))
-DP algorithm M that takes n private samples and m public samples from D ∈ G ( d, k, s ) , and is private with respect to the private samples, such that if D = { ( µ 1 , Σ 1 , w 1 ) , . .
, ( µ k , Σ k , w k ) } satisfies Condition 2, and
[Equation]
then M ( α, β ) -learns D .
Proof.
We mainly focus on the privacy guarantees here because the accuracy would follow from
Theorem C.22 after setting the parameters appropriately.
Note that the privacy parameters in this case in the set PrivParams are ε, δ .
As per the advanced composition guarantees of DP (Lemma A.19), we set the privacy parameters in PrivParams ′ in Algorithm 6 to be ε ′ =
ε √ 12 k
log(1 /δ ) and δ ′ =
δ 4 k .
Define ε 0 = √ kε ′ .
Now, we show that the intermediate steps in the call to Algorithm 6 yield ( ε, δ ) -DP.
We don’t release the output of that step itself, but use it in subsequent steps.
The clusters are formed on the basis of intersection of the privately formed balls with the private dataset, and are then used subsequently in other private algorithms.
So, the final algorithm would be private, as well.
Thus, it is enough to prove the privacy guarantees of those intermediate steps that yield the said balls.
In Algorithm 6, we start working on the private dataset X inside the loop, specifically, when we use private PCA.
In each iteration i , we ensure that each row of the input to Algorithm( Z i − M i ) has norm at most R i by virtue of selecting points that lie in a ball of radius R i , and by recentering the points appropriately.
Then by Lemma C.19, the sensitivity of Y T Y in Algorithmis at most 2 R 2 i .
Therefore, adding Gaussian noise calibrated to ε ′ , δ ′ to Y T Y , that is, setting f PCA ( ε ′ , δ ′ ) =
[Equation]
, implying that σ
[Equation]
ε ′ is enough to ensure ( ε ′ , δ ′ )
-DP in this step (Lemmata A.22 and A.18).
The next step in Algorithm 6 that works with the private data is at the call to Algorithm 5.
In this algorithm, only one call to PCount is performed, and this is the only time it works with the private dataset.
Therefore, by the guarantees of Lemma A.23, we know that this step is ε ′
-DP.
The next steps are either partitioning the private data or adding the isolated private data to C .
Since neither of those sets are being released to public, there is no privacy loss here.
Therefore, each iteration is (2 ε ′ , δ ′ )
-DP (by Lemma A.19).
Applying composition over all 2 k iterations, we have that all operations together in the entire run of the loop in the algorithm are (2 ε, δ ) -DP.
In Theorem C.21, setting f ′ PCA (PrivParams) =
f PCA (
[Equation] [Equation]
0 ) ∈
[Equation]
) ε (cid:17) give us the right sample complexity for accuracy for all calls to Algorithm 4 (by Lemma A.19), since we already multiply the √ k factor in the numerator.
Similarly, setting f ′ PCount (PrivParams) =
f PCount ( ε 0 ) = ε √ 12log(
/δ
in Theorem C.21 gives us the right sample complexity for all calls to Algorithm 5.
Now, each call to the approximate-DP Gaussian learner (Lemma A.24) is on a disjoint part of the private dataset X .
Therefore, all the k calls together are ( ε, δ ) -DP because changing one point in X can change one point in only one of the clusters.
By the same reasoning, all calls to PCount together are ( ε, δ ) -DP.
As far as the accuracy goes, the first two terms in the sample complexity ensures that enough points go to each call to the Gaussian estimator from Lemma A.24 and to each call to PCount because of
Lemma C.7.
Therefore, our GMM estimator ( α, β ) -learns the mixture.
C.2.2 ρ -zCDP Algorithm
Here, we state our results under zCDP, and instantiate Algorithms 4, 6, and 7 for this version of DP.
Theorem C.24.
For all α, β, ρ > 0 , there exists an O ( ρ ) -zCDP algorithm M that takes n private samples and m public samples from D ∈ G ( d, k, s ) , and is private with respect to the private samples, such that if D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } satisfies Condition 2, for each i ∈ [ k ] , ∥ µ i ∥ ≤ R and I ⪯ Σ
i ⪯ K I , and
[Equation]
  m
∈ O (cid:18) log( k/
[Equation]
w min (cid:19) , then M ( α, β ) -learns D .
Proof.
Again, we focus on the privacy guarantees because the accuracy would follow from Theorem C.22 after setting the parameters correctly.
Note that the privacy parameter in this case in the set PrivParams is ρ .
As per the advanced composition guarantees of DP (Lemma A.19), we set the privacy parameter in PrivParams ′
in Algorithm 6 to be ρ ′ =
ρ 2 k .
Now, we show that the intermediate steps in the call to Algorithm 6 yield O ( ρ )
-zCDP.
As before, it is enough to prove the privacy guarantees of those intermediate steps, since the output of the algorithm itself wouldn’t be released, but would be used by Algorithm 7, instead.
In Algorithm 6, we start working on the private dataset X inside the loop, specifically, when we use private PCA.
In each iteration i , by the same argument as in the proof of Theorem C.23, the sensitivity of Y T Y in the call to Algorithm 4 is 2 R 2 i .
Therefore, adding Gaussian noise calibrated to ρ ′ to Y T Y , that is, setting f PCA ( ρ ′ )
= 1 √ 2 ρ ′ , implying that σ P = 2 R 2 i √ 2 ρ ′ is enough to ensure ρ ′
-zCDP in this step (Lemmata A.22 and A.18).
The next step in Algorithm 6 that works with the private data is at the call to Algorithm 5.
In this algorithm, only one call to PCount is performed, and this is the only time it works with the private dataset.
Therefore, by the guarantees of Lemma A.23, we know that this step is ρ ′
-zCDP.
Therefore, each iteration is 2 ρ ′
-zCDP (by Lemma A.19).
Applying composition over all 2 k iterations, we have that all operations together in the entire run of the loop in the algorithm are 2 ρ -zCDP.
In Theorem C.21, setting f ′ PCA (PrivParams) =
f PCA ( ρ ) ∈
O (cid:16) 1 √ ρ (cid:17) give us the right sample complexity for accuracy for all calls to Algorithm 4 (by Lemma A.19), since we already multiply the √ k factor in the numerator.
Similarly, setting f ′ PCount (PrivParams) =
f PCount ( √ 2 ρ ) = √ 2 ρ in Theorem C.21 gives us the right sample complexity for all calls to Algorithm 5.
Now, each call to the zCDP
Gaussian learner (Lemma A.24) is on a disjoint part of the private dataset
X .
Therefore, all the k calls together are ρ -zCDP because changing one point in X can change one point in only one of the clusters.
By the same reasoning, all calls to PCount together are ρ -zCDP.
For the accuracy guarantees, the first two terms in the sample complexity ensures that enough points go to each call to the Gaussian estimator from Lemma A.24 and to each call to PCount because of
Lemma C.7.
Therefore, our GMM estimator ( α, β ) -learns the mixture.
C. (cid:101) O ( d/w min ) Public Samples
In this subsection, we assume that we have (cid:101) O
( d/w min ) public data samples available.
In this case, the number of public samples is enough to do PCA accurately.
So, the approach would be much simpler than before, since we don’t need to find a supercluster anymore.
Also, because we have enough public samples, we could simply use the public dataset itself to partition the data when in low dimensions.
Therefore, the entire clustering operations to partition the private dataset could be done using the public dataset itself, without having to touch the private dataset at all.
As in the previous subsection, we will first give a general private algorithm for estimating the GMM, but will instantiate it separately for approximate DP and zCDP.
We start with the low-dimensional partitioner first, which just uses the public dataset, and the public query, as defined in 3.
, m ′ c
j ← (cid:101) Y j
If Q Pub ( (cid:101) Y , c j , r i , mw min)
=
T RUE
Return j , R
2 r i )
Return
Theorem C.25.
Let Y be a clean subset of a set of public samples from D ∈ G ( d, k ) , Π be a projection matrix to ℓ dimensions, (cid:101)
Y =
Y Π be the input to Algorithm 8, r max , r min > 0 , such that for some j ∈ [ m ′ ] , (cid:101) Y ∩ B r max (cid:16)(cid:101) Y i (cid:17) =
(cid:101) Y and B r max (cid:16)(cid:101) Y i (cid:17) is pure with respect to the components of
D projected on to the subspace of Π having points in (cid:101) Y .
Suppose σ 2max is the largest directional variance among the Gaussians that have points in Y .
Let that Gaussian be N ( µ, Σ) .
Suppose, m ≥ O (cid:18)
d ln( k/β ) w min (cid:19) .
Then we have the following with probability at least 1 − β .
Suppose there are at least two Gaussians that have points in Y .
For any other Gaussian
N ( µ ′ , Σ ′ ) that has points in Y , suppose for N > m , ∥ µ Π − µ ′ Π
∥ ≥ Ω( σ max (cid:112) ℓ ln( Nℓ/β )) .
Let r min ≤ σ max (cid:112) 2 ℓ ln(2 Nℓ/β ) .
Then the ball returned by Algorithm 8 is pure with respect to the components of D projected by Π , which have points in (cid:101) Y , such that R d \ B R ( c ) only contains components from D projected by Π that have points in (cid:101) Y , and so does B R ( c ) .
Suppose there is only one Gaussian that has points in Y .
Then the algorithm returns ⊥ .
Proof.
The proof for the first part is the same as that for the first part of Theorem C.20.
So, we don’t discuss that any further.
For the next part, note that we could relax the query Q Pub by modifying the second constraint to say | X ∩ ( B 11 r ( c ) \ B r ( c ))
| < t 320 .
If the answer to the original query is true, then the answer to the modified query would be true, as well.
In other words, if the answer to the modified query is false, then the answer to the original query is false, too.
Therefore, it is enough to show that with probability at least 1 − β , there cannot be an 11 -terrific ball in (cid:101) Y .
As argued in the proof of Theorem C.20, it is enough to show that there exists no 3 -terrific ball in (cid:101) Y .
We already proved this in the proof of
Theorem C.20.
Hence, we have the claim.
Next, we provide a clustering algorithm to partition the private dataset that just operates on the public dataset (Algorithm 9).
The algorithm is the same as Algorithm 6, but uses the public data solely to partition the private data.
It uses tools like PCA and Algorithm 8 on public data.
Let Y ← (cid:101)
X and Z ← X .
Let C ← ∅ .
Let Q Priv and Q Pub be queues of sets of points.
Add set Z to Q Priv , and set Y to Q Pub .
Set count ← 0 and i ← 1 .
While count < k and i ≤ 2 k
Pop Q Priv to get Z , and Q Pub to get Y .
// PCA: project points of both datasets onto returned subspace.
Let Π i be the top- k subspace of Y T Y .
Y ′
← Y Π i and Z ′ ← Z Π i .
// If there is only one Gaussian, add it to C , otherwise further partition the datasets.
Let R i ← 4 max y 1 ,y 2 ∈ Y {∥ y 1 − y 2 ∥} and r i ← √ 2 k ln(2( n + m ) k/β ) 4 √ d min y 1 ,
y 2 ∈ Y {∥ y 1 − y 2 ∥} .
B ← LowDimPartitioner m,w min ( Y ′ , R i , r i ) .
If B = ⊥
C ← C ∪ { Z i } .
count ← count + 1 .
Else
B is an ordered pair ( c ′ i , r ′ i ) .
S ′ ← B
r
i ( c ′ )
∩ Y ′ and T
′ ← B r
i ( c ′ )
∩ Z ′ .
Let S be points in Y i corresponding to S ′ , and T be points in Z i corresponding to T ′ .
Add S to Q Pub and T to Q Priv .
Add Y i \ S to Q Pub and Z i \ T to Q Priv .
i ← i +
Return C .
Theorem C.26.
There exists an algorithm (Algorithm 9) that takes n private samples X , and m public samples from D ∈ G ( d, k, s ) satisfying Condition 2, and outputs a partition C of X , such that given Conditions C.1 to C.6 hold, and
[Equation]
and m ≥ O (cid:18)
[Equation]
β
[Equation]
w
[Equation]
(cid:19) , then with probability at least 1 − O ( β ) , | C |
=
k and each S ∈ C is clean and non-empty.
Proof.
As in the proof of Theorem C.21, it is enough to show that ∀ i ≥ 0 , at the end of the i -th iteration, (1) a new Gaussian component is isolated in C and count equals the number of isolated components in C , (2) or the private dataset X is further partitioned into non-empty, clean subsets, and the public dataset (cid:101) X is further partitioned into clean subsets, such that | Q Pub | = | Q Priv | and for each j ∈ [ | Q Pub | ] , Q Pub [ j ] and Q Priv [ j ] have points from the same Gaussian components.
We can prove this via induction on i .
Base Case: The proof is exactly the same as in the proof of Theorem C.21.
Inductive Step: We assume for all iterations, up to and including some i ≥ 0 , the claim holds.
Then we show that it holds for iteration i + 1 , as well.
By the inductive hypothesis, we know that Z and Y are clean subsets, and contain points from the same components.
We first reason about the correctness of the PCA step.
Using Lemma A.7 and our bound on m , we know that for each component ( µ ′ , Σ ′ , w ′ ) , the empirical mean of all the points from that Gaussian in (cid:101)
X would be at most
[Equation]
w
[Equation]
away from µ ′ .
By the same reasoning, the empirical mean of the same projected points would be at most
[Equation]
away from µ ′ Π because the projection of a Gaussian is still a Gaussian.
Lemmata A.13 (by setting B = 0 ) and A.14 together guarantee that the distance between the empirical mean of those points and that of the projected points of that component would be at most O (cid:16) σ max √ w ′ (cid:17) .
The triangle inequality implies that ∥ µ ′ − µ ′ Π ∥ ≤ O (cid:16) σ max √ w ′ (cid:17) .
This happens with probability at least 1 − β 2 k .
Now, if there is just one component that has points in Y (hence, in Z ), then on feeding Y ′ to
Algorithm 5, we will get ⊥ (by the guarantees from Theorem C.20) with probability at least 1 − β 2 k .
Because Z is a clean subset, C now contains one more clean subset of X , and we have isolated a new component, and updated the value of count to reflect the change.
Suppose there are at least two components that have points in Y (hence, in Z ).
Let the largest directional variance among the Gaussians that have points in Y be σ 2max , its mean be µ , and its mixing weight be w .
From the guarantees of the PCA step, we know that ∥ µ − µ Π i ∥ ≤ O (cid:16) σ max √ w (cid:17) .
Because of the separation condition, applying the triangle inequality, we know that the distance between µ Π i and the projected mean of any other Gaussian having points in Z Π is at least Ω( σ max (cid:112) k
ln(( n + m ) k ) /β ) .
Also, we know by the construction of the algorithm that for any y ∈ Y ,
Y ⊆
B R i
( y ) , and from Lemmata C.8, C.9 and C.11 that B R i ( y ) would contain all the components of D that have points in Y .
Also, by Lemmata C.9, and C.11, r i ≤ σ max (cid:112) 2 k ln(2( n + m ) k/β ) .
Then from the guarantees of Theorem C.25, we have that a ball is returned, and it is pure with respect to the components of the mixture projected by Π i that have points in Y ′ .
Therefore, subsets S ′
and
T ′ are clean, and so are S and T .
This implies that Y \ S and Z \ T are clean, too.
By adding them in order to Q Pub and Q Priv respectively, we partition the datasets (cid:101) X and X further into clean subsets, and preserve the ordering of the components in the respective queues.
This proves the claim.
As argued before in the proof Theorem C.21, allowing up to 2 k iterations is enough.
Therefore, we get a clean partition of X .
Finally, we provide the DP algorithm for learning GMM’s in this regime of public data.
The algorithm is the same as Algorithm 7, except that it calls Algorithm 9, instead of Algorithm 6.
Theorem C.27.
For all α, β > 0 and sets of privacy parameters PrivParams , there exists an algorithm (Algorithm 10) that takes n private samples and m public samples from D ∈ G ( d, k, s ) , such that if D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } satisfies Condition 2, and
[Equation]
where n GE is the sample complexity of privately learning a Gaussian using Lemma A.24 and
Theorem B.5 according to the type of DP required, then it ( α, β ) -learns D .
Input: Private samples X
=
( X 1 , . . .
X n ) ∈ R n × d .
Public samples (cid:101) X
=
( (cid:101) X 1 , . . .
, (cid:101)
X m ) ∈ R m × d .
Parameters PrivParams ⊂ R , β, k, w min > 0 .
Output: Set (cid:98) D
= { ( (cid:98) µ 1 , (cid:98) Σ 1 , (cid:98) w 1 ) , . . .
, ( (cid:98) µ k , (cid:98) Σ k , (cid:98) w k ) } .
// Clustering.
Set C ← EasyClustering β,k,w min ( X, (cid:101) X ) .
Parameter estimation.
(cid:98) D ← ∅ .
For i ← 1 , . . .
, k
Set ( (cid:98) µ i , (cid:98) Σ i ) ← DPGaussianEstimator PrivParams , βk ( C i ) .
Set (cid:98) w i ← max (cid:8) 1 n · PCount PrivParams ( | C i | ) , α 2 k (cid:9) .
(cid:98) D ← (cid:98) D ∪ { ( (cid:98) µ i , (cid:98) Σ
i , (cid:98) w i ) } .
Return (cid:98) D .
Proof.
By Theorem C.26, we know that with probability at least 1 − β 3 , | C | =
k , and for each i ∈ [ k ] ,
C i is a clean subset of X .
Next, by the guarantees of Lemma A.24 and Theorem B.5, for each i ∈ [ k ] , with probability at least 1 − β 3 k , d TV ( N ( (cid:98) µ i , (cid:98) Σ i ) , N ( µ i , Σ i )) ≤ α .
Finally, by the same argument as in the proof of Theorem C.22, the mixing weights are estimated accurately, as well.
Therefore, taking the union bound over all failure events, we have the required result.
C.3.1 ( ε, δ ) -DP Algorithm
Here, we instantiate Algorithm 10 for the case of approximate DP.
Theorem C.28.
For all α, β, ε, δ > 0 , there exists an ( O ( ε ) , O ( δ ))
-DP algorithm M that takes n private samples and m public samples from D ∈ G ( d, k, s ) , and is private with respect to the private samples, such that if D = { ( µ 1 , Σ 1 , w 1 ) , . .
, ( µ k , Σ k , w k ) } satisfies Condition 2, and
[Equation]
m
[Equation]
then M ( α, β ) -learns D .
Proof.
We again focus on the privacy guarantees here because the accuracy would follow from
Theorem C.27 after setting the parameters appropriately.
Note that the privacy parameters in this case in the set PrivParams are ε, δ .
We just have to show the privacy guarantees for the calls to the
Gaussian estimator and PCount because all other steps involve computations on the public dataset.
By the same argument as in the proof for Theorem C.23, all k calls together to the approximate DP learner from Lemma A.24 are ( ε, δ ) -DP, and all calls together to PCount are ( ε, δ ) -DP.
Therefore, we get (2 ε, 2 δ ) -DP guarantee in the end.
For the accuracy goal, the sample complexity ensures that enough points go to each call to the
Gaussian estimator from Lemma A.24 and to each call to PCount because of Lemma C.7.
Therefore, our GMM estimator ( α, β ) -learns the mixture.
C.3.2 ρ -zCDP Algorithm
Finally, we instantiate Algorithm 10 for the case of zCDP.
Note that we don’t need any bounds on the range parameters in this case because we are using the Gaussian learner from Section B.
Theorem C.29.
For all α, β, ρ > 0 , there exists an O ( ρ ) -zCDP algorithm M that takes n private samples and m public samples from D ∈ G ( d, k, s ) , and is private with respect to the private samples, such that if D = { ( µ 1 , Σ 1 , w 1 ) , . . .
, ( µ k , Σ k , w k ) } satisfies Condition 2, and
[Equation]
then M ( α, β ) -learns D .
Proof.
We mainly prove the privacy guarantees because the accuracy would follow from Theorem C.27 after setting the parameters correctly.
Note that the privacy parameter in this case in the set
PrivParams is ρ .
Using the same argument as in the proof of Theorem C.23, we have ρ -zCDP for all k calls to the learner in Section B (by Theorem B.5), and ρ -zCDP for all calls together to PCount .
Therefore, we have 2 ρ -zCDP in the end.
By the same argument as before, the sample complexity is enough to send enough points to each call to the learner in Section B and to PCount .
Therefore, our GMM estimator ( α, β ) -learns the mixture.
D A Proof-of-Concept Numerical Result
For Gaussian mean estimation, our approach is relatively simple to implement on top of an existing private algorithm.
We offer some proof-of-concept simulations that demonstrate the effectiveness of public data in private statistical estimation.
In Figure 1, we show plots that evaluate 1-public-sample private mean estimation (the algorithm described in Section 2.1 and in more detail in Appendix
B.1.1).
We examine the effect of 1 public sample on the performance of CoinPress [BDKU20] with its best parameter setting ( t = 2 ), in a case where the initial a priori bounds on the mean are weak.
Concretely, we draw n samples from a d = 50 dimensional Gaussian N ( µ k , I d ) , where µ k = k · [1 , ..., 1] T and correspondingly set our a priori bound R = k √ d for CoinPress.
We show results for k ∈ [10 , 100 , 1000] , representing varying levels of strength in our a prior bounds on the mean.
We follow the evaluation protocol from [BDKU20]: we target zCDP
with ρ = 0 . 5 , and at each sample size n we run the estimator 100 times and report the 10% trimmed mean of error from the ground truth (we additionally report the 10% trimmed standard deviation as error bars).
We also follow their practice of treating target failure probabilities β i of various steps of the algorithm as hyperparameters that can be tuned for the best empirical results.
The new step we introduce (using the public sample to set R based on d, β ) uses β = 0 .
01 , which is the same value used for all the β i ’s in CoinPress.
The numerical result demonstrates the promise of utilizing public data for private data analysis, and confirms the takeaway that very little public data can help greatly when a priori knowledge of the private data is weak.
As is visible from the plots, the error of our public-private algorithm tracks the non-private algorithm closer when a priori bounds on the mean are weak ( k = 1000 case).
Note that these results are only meant to be a proof-of-concept simulation to demonstrate the promise of public data – thorough tuning and evaluation of these algorithms (which is necessary to bring these algorithms to practice) is an important direction for future work.
[Figure]