Re-Analyze Gauss: Bounds for Private Matrix Approximation via Dyson Brownian Motion
Abstract
Given a symmetric matrix M and a vector ⁄ , we present new bounds onthe Frobenius-distance utility of the Gaussian mechanism for approximating M by a matrix whose spectrum is ⁄ , under ( Á , ” )
-di  erential privacy.
Our bounds depend on both ⁄ and the gaps in the eigenvalues of M , andhold whenever the top k + 1 eigenvalues of M have su  ciently large gaps.
When applied to the problems of private rank- k covariance matrix approximation and subspace recovery, our bounds yield improvements over previous bounds.
Our bounds are obtained by viewing the addition of Gaussian noise as a continuous-time matrix Brownian motion.
This viewpointallows us to track the evolution of eigenvalues and eigenvectors of the matrix, which are governed by stochastic di  erential equations discovered by
Dyson.
These equations allow us to bound the utility as the square-root ofa sum-of-squares of perturbations to the eigenvectors, as opposed to a sumof perturbation bounds obtained via Davis-Kahan-type theorems.
Introduction
Given a dataset A œ R m ◊ d , which consists of m individuals with d -dimensional features,methods for preprocessing or prediction from A often use the covariance matrix M :=
A € Aof A .
In many such applications one computes a rank- k approximation to M , or ﬁnds amatrix close to M with a speciﬁed set of eigenvalues ⁄
= ( ⁄ 1 , . .
. , ⁄ d ) [38, 28, 37].
Examplesinclude the rank- k covariance matrix approximation problem where one seeks to compute arank- k matrix which minimizes a given distance to M and the subspace recovery problemwhere the goal is to compute a rank k -projection matrix H =
V k V € k , where V k is thed ◊ k matrix whose columns are the top- k eigenvectors of M .
These matrix approximationproblems are ubiquitous in ML and have a rich algorithmic history; see [29, 46, 10, 8].
In some cases, the rows of A correspond to sensitive features of individuals and the releaseof solutions to aforementioned matrix approximation problems may reveal their privateinformation, e.g., as in the case of the Netﬂix prize problem [5].
Di  erential privacy (DP)has become a popular notion to quantify the extent to which an algorithm preserves privacyof individuals [15].
Algorithms for solving low-rank matrix approximation problems have been widely studied under DP constraints [30, 7, 19, 17].
Notions of DP studied in theliterature include ( Á , ” )
-DP [17, 25, 26, 19] which is the notion we study in this paper,as well as pure ( Á , 0) -DP
To deﬁne a notion of DP in problems involvingcovariance matrices, following [7, 17], two matrices M =
A € A and M Õ =
A Õ€ A Õ are said to be neighbors if they arise from A, A Õ which di  er by at most one row and as, is oftentimesdone, require that each row of the datasets A, A Õ has norm at most 1 .
For any Á , ” Ø 0 , a randomized mechanism A is ( Á , ” ) -di  erentially private if for all neighbors M, M Õ
œ R d ◊ d , and measurable
P e
P ( A ( M Õ )
S )
The problem.
We consider a class of problems where one wishes to compute an approximation to a symmetric d ◊ d matrix under ( Á , ” )
-di  erential privacy constraints.
Speciﬁcally,given M =
A € A for A œ R m ◊ d , together with a vector ⁄ of target eigenvalues ⁄ 1 Ø · · · Ø ⁄ d ,the goal is to output a d ◊ d matrix ˆ H with eigenvalues ⁄ which minimizes the Frobenius-normdistance Î ˆ H ≠ H Î F under ( Á , ” )
-di  erential privacy constraints.
Here H is the matrix witheigenvalues ⁄
and the same eigenvectors as M .
This class of problems includes as a specialcase the subspace recovery problem if we set ⁄ 1 = · · · = ⁄ k
= 1 and ⁄ k +1 = · · · = ⁄ d = 0 .
It also includes the rank- k covariance approximation problems if we set ⁄
i =
i for i Æ k , 45 where ‡Ø · · · Ø ‡ d are the eigenvalues of M .
Since revealing ‡ i s may violate privacyconstraints, the eigenvalues of the output matrix ˆ H should not be the same as those of H .
Various distance functions have been used in the literature to evaluate the utility of ( Á , ” ) -
DP mechanisms for matrix approximation problems, including the Frobenius-norm distanceÎ ˆ H ≠ H Î F (e.g. [19, 2])and the Frobenius inner product utility È M, H ≠
ˆ H Í (e.g. [11, 19, 24]).
Note that while a bound Î H ≠
ˆ H Î F Æ b implies an upper bound on the inner product utilityof È M, H ≠
ˆ H Í Æ Î M Î F · b (by the Cauchy-Schwarz inequality), an upper bound on the 52 inner product utility does not (in general) imply any upper bound on the Frobenius-normdistance.
Moreover, the Frobenius-norm distance can be a good utility metric to use ifthe goal is to recover a low rank matrix H from a dataset of noisy observations (see e.g.[12]).
Hence, we use the Frobenius-norm distance to measure the utility of an ( Á , ” ) -DPmechanism.
Related work.
The problem of approximating a matrix under di  erential privacy constraints has been widely studied.
In particular, prior works have provided algorithms forproblems where the goal is to approximate a covariance matrix under di  erential privacyconstraints, including rank- k PCA and subspace recovery [7, 30, 19, 33] as well as rank- kcovariance matrix approximation [7, 19, 2].
Another set of works have studied the problemof approximating a rectangular data matrix A under DP [7, 1, 25, 26].
We note that upperbounds on the utility of di  erentially-private mechanisms for rectangular matrix approximation problems can grow with the number of datapoints m , while those for covariancematrix approximation problems oftentimes depend only on the dimension d of the covariancematrix and do not grow with m .
Prior works which deal with covariance matrix approximation problems such as rank- k covariance matrix approximation and subspace recoveryare the most relevant to our paper.
The notion of DP varies among the di  erent workson di  erentially-private matrix approximation, with many of these works considering thenotion ( Á , ” )
-DP [25, 26, 19], while other works focus on (pure) ( Á , 0) -DP [30, 2, 33].
Analysis of the Gaussian mechanism in [19].
[19] analyze a version of the Gaussian mechanism of [16], where one perturbs the entries of M by adding a symmetric matrix E with iid
Gaussian entries N (0 , Ô log( 1 ” ) / Á ) , to obtain an ( Á , ” ) -di  erentially private mechanism whichoutputs a perturbed matrix ˆ M = M + E .
One can then post-process this matrix ˆ M toobtain a rank- k projection matrix which projects onto the subspace spanned by the top- keigenvectors of ˆ M (for the rank- k PCA or subspace recovery problem), or a rank- k matrixˆ H with the same top- k eigenvectors and eigenvalues as ˆ M (for the rank- k covariance matrixapproximation problem).
[19] consider di  erent notions of utility in their results, includingthe inner product utility (for PCA), and the Frobenius-norm and spectral-norm distancedistances (for low-rank approximation and subspace recovery).
In one set of results, [19] give lower utility bounds of ˜  ( k Ô d ) w.h.p.
for the rank- k PCAproblem with respect to the inner product utility È M, H Í , together with matching upperbounds provided by a post-processing of the Gaussian mechanism, where ˜  hides polynomialfactors of 1 Á and log( 1 ” ) (their Theorems 3 and 18).
As noted by the authors, their lowerbounds are tight for matrices M with the “worst-case”
spectral proﬁle
‡ , but they can obtainimproved upper bounds for matrices M where
‡ k ≠ ‡ k +1 > ˜  ( Ô d ) (Theorem 3 of [19]).
For the subspace recovery problem, [19] obtain a Frobenius-distance bound of Î ˆ H
≠ H Î F Æ˜ O !
Ô kd / ( ‡ k ≠ ‡ k +1 ) " w.h.p.
for a post-processing of the Gaussian mechanism whenever‡ k ≠ ‡ k +1 > ˜  ( Ô d ) (implied by their Theorem 6, which is stated for the spectral norm).
And for the rank- k covariance matrix approximation problem, [19] show a utility bound of
Î ˆ H ≠ M Î F ≠Î H ≠ M Î F Æ ˜ O
( k Ô d ) w.h.p.
for a post-processing of the Gaussian mechanism 92 (Theorem 7 in [19]), and also give related bounds for the spectral norm.
While their Frobenius bound for the covariance matrix approximation problem is independent of the numberof datapoints m , it may not be tight.
For instance, when k = d , one can easily obtain abetter bound since, by the triangle inequality, Î ˆ H ≠ M Î F ≠ Î H ≠ M Î F Æ Î ˆ H
≠ H Î F =Î ˆ M ≠ M Î F = Î E Î F Æ O ( d ) w.h.p.
, since Î E Î F is just the norm of a vector of d 2 Gaussians with variance
˜ O (1) .
Moreover, the bound for the the rank- k covariance approximationproblem, Î ˆ H ≠ H Î F Æ ˜ O
( k Ô d ) , is also a worst-case upper bound for any spectral proﬁle ‡as
the right hand side of the bound not depend on the eigenvalues ‡ .
Thus, a question arises of whether the Frobenius-norm utility bounds for the rank- k covariance matrix approximation and subspace recovery problems are tight for all spectral proﬁles‡ , and whether the analysis of the Gaussian mechanism can be improved to achieve better utility bounds.
A more general question is to obtain utility bounds for the Gaussianmechanism for the matrix approximation problems for arbitrary ⁄ .
Our contribution.
Our main result is a new upper bound on the Frobenius-distance utilityof the Gaussian mechanism for the general matrix approximation problem for a given Mand ⁄
(Theorem 2.2).
Our bound depends on the eigenvalues of M and the entries of ⁄ .
The novel insight is to view the perturbed matrix M + E as a continuous-time symmetricmatrix di  usion, where each entry of the matrix M + E is the value reached by a (onedimensional) Brownian motion after some time T =
Ô log( 1 ” ) / Á .
This matrix-valued Brownianmotion, which we denote by  ( t ) , induces a stochastic process on the eigenvalues “ 1 ( t ) Ø· · · Ø “ d ( t ) and corresponding eigenvectors u 1 ( t ) , . .
, u d ( t ) of  ( t ) originally discovered by
Dyson and now referred to as Dyson Brownian motion, with initial values “ i (0) =
‡ i andu i (0) which are the eigenvalues and eigenvectors of the initial matrix M [20].
We then use the stochastic di  erential equations (2) and (3), which govern the evolution ofthe eigenvalues and eigenvectors of the Dyson Brownian motion, to track the perturbationsto each eigenvector.
Roughly speaking, these equations say that, as the Dyson Brownianmotion evolves over time, every pair of eigenvalues “ i ( t ) and “ j ( t ) , and corresponding eigenvectors u i ( t ) and u j ( t ) , interacts with the other eigenvalue/eigenvector with the magnitudeof the interaction term proportional to 1 “ i ( t ) ≠ “ j ( t ) at any given time t .
This allows us tobound the perturbation of the eigenvectors at every time t , provided that the initial gaps inthe eigenvectors ‡ i ≠ ‡ j are su  ciently large.
We then derive a stochastic di  erential equation which tracks how the utility changes as the Dyson Brownian motion evolves over time(Lemma 4.1), and integrate this di  erential equation over time to obtain a bound on the(expectation of) the utility E [ Î ˆ H ≠ H Î F ] (Lemma 4.5) as a function of the gaps “ i ( t ) ≠ “ j ( t ) .
Plugging in basic estimates (Lemma 4.4) for the eigenvalue gaps “ i ( t ) ≠ “ j ( t ) to Lemma4.5, we obtain a bound on the expected utility
E
[ Î ˆ H ≠ H Î F ] (Theorem 2.2) for the di  erentmatrix approximation problems as a function of the eigenvalue gaps ‡ i ≠ ‡
j of the inputmatrix M .
Roughly speaking, our bound is the square-root of a sum-of-squares of the ratios,⁄ i ≠ ⁄ j ‡ i ≠ max( ‡ j ,
‡ k +1 ) , of eigenvalue gaps of the input and output matrices.
When applied to the rank- k covariance matrix approximation problem (Corollary 2.3), Theorem 2.2 implies a bound of E [ Î ˆ H ≠ H Î F ] Æ ˜ O
( Ô kd ) whenever the eigenvalues ‡ of theinput matrix M satisfy ‡ k ≠ ‡ k +1 Ø  ( ‡ k ) and the gaps in top k + 1 eigenvalues satisfy ‡ i ≠
‡ i +1 Ø ˜  ( Ô d ) .
Thus, when M satisﬁes the above condition on ‡ , our boundimproves by a factor of Ô k on the (expectation of) the previous bound of [19], whichsays that Î ˆ H ≠ M Î F ≠ Î H ≠ M Î F Æ ˜ O
( k Ô d ) w.h.p., since by the triangle inequalityÎ ˆ H ≠ M Î F ≠ Î H ≠ M Î F Æ Î ˆ H
≠ H Î F .
This condition on ‡ is satisﬁed, e.g., for ma- 138 trices M whose eigenvalue gaps are at least as large as those of the Wishart random covariance matrices with su  ciently many datapoints m (see Section 2 for details).
And,if ‡ is such that ‡ i ≠ ‡ i +1 Ø  ( ‡ k
≠ ‡ k +1 ) for i Æ k , Theorem 2.2 implies a bound of
E [ Î ˆ H ≠ H Î F ] Æ ˜ O
( Ô d / ( ‡ k ≠ ‡ k +1 ) ) for the subspace recovery problem (Corollary 2.4), improving by a factor of Ô k (in expectation) on the previous bound of [19], which implies that
[Equation]
≠ ‡ k +1 ) " w.h.p.
Results
Our main result (Theorem 2.2) gives a new and uniﬁed upper bound on the Frobenius-normutility of a post-processing of the Gaussian mechanism, for the general matrix approximationproblem where one is given a symmetric matrix M œ R d ◊ d and a vector ⁄ with ⁄ 1 Ø · · · Ø ⁄ d ,and the goal is to compute a matrix ˆ H with eigenvalues ⁄ which minimizes the distanceÎ ˆ H ≠ H Î F .
Here H is the matrix with eigenvalues ⁄
and the same eigenvectors as M .
Plugging in di  erent choices of ⁄ to Theorem 2.2, we obtain as corollaries new Frobeniusdistance utility bounds for the rank- k covariance matrix approximation problem (Corollary2.3) and the subspace recovery problem (Corollary 2.4).
Our results rely on the followingassumption about the eigenvalues of the input matrix M :
Assumption 2.1 (( M, k, ⁄ 1 , Á , ” )
Eigenvalue gaps) .
The gaps in the top k + 1 eigenvalueseigenvalues ‡ 1 Ø · · · Ø ‡ d of the matrix M œ R d ◊ d satisfy ‡ i ≠ ‡
[Equation]
+1 Ø 8 Ô log(
[Equation]
Á Ô d +3 log 12 ( ⁄ k ) for every i œ [ k ] .
Assumption 2.1 is satisﬁed, for instance, by random Wishart matrices W
=
A € A , where Ais an m ◊ d matrix of iid Gaussian entries, which are a popular model for sample covariancematrices [48].
This is because the minimum gap ‡ i ≠ ‡ i +1 of a Wishart matrix growsproportional to Ô m with high probability; thus for large enough m , Assumption 2.1 holds(see Appendix I for details).
Hence, the assumption requires that the gaps in the top k + 1eigenvalues of M are at least as large as the gaps in a random Wishart matrix.
Theorem 2.2 ( Main result). Let Á , ” > 0 , and given a symmetric matrix M œ R d ◊ dwith eigenvalues ‡ 1 Ø · · · Ø ‡ d and corresponding orthonormal eigenvectors v 1 , . .
. , v d .
Let G be a matrix with iid N (0 , 1) entries, and consider the mechanism that outputs ˆ M =
M + Ô 2log( 1 . 25 ” )
Á ( G + G € ) .
Then such a mechanism is ( Á , ” ) -di
erentially private.
Moreover,let ⁄ 1 Ø · · · Ø ⁄ d and k œ [ d ] be any numbers such that ⁄
i =
for i > k , and deﬁne := diag( ⁄ 1 , . . .
, ⁄ d ) and V =
[ v 1 , . .
. , v d ] , and deﬁne ˆ ‡ 1 Ø · · · Ø ˆ ‡ d to be the eigenvaluesof ˆ M with corresponding orthonormal eigenvectors ˆ v 1 , . . .
, ˆ v d and ˆ V =
[ˆ v 1 , . . .
, ˆ v d ] .
Then if
M satisﬁes Assumption 2.1 for ( M, k, ⁄ 1 , Á , ” ), we have
[Equation]
The fact that the mechanism in this theorem is ( Á , ” ) -di  erentially private follows from standard results about the Gaussian mechanism [19].
Given any list of eigenvalues ⁄ , and letting = diag( ⁄ ) , one can post-process the matrix ˆ M by computing its spectral decompositionˆ M =
ˆ V ˆ  ˆ V € and replacing its eigenvalues to obtain a matrix
ˆ V  ˆ V € with eigenvalues ⁄ andeigenvectors ˆ V .
Since ˆ V  ˆ V € is a post-processing of the Gaussian mechanism, the mechanism which outputs ˆ V  ˆ V € is di  erentially private as well.
Theorem 2.2 bounds the excessutility
E [ Î ˆ V
ˆ V € ≠ V  V € Î 2 F ] (whenever the gaps in the eigenvalues ‡ 1 Ø · · · Ø ‡ d of theinput matrix satisfy Assumption 2.1) as a sum-of-squares of the ratio of the gaps ⁄
i ≠ ⁄ jin the given eigenvalues to the corresponding gaps ‡ i ≠ max( ‡ j , ‡ k +1 ) in the eigenvalues ofthe input matrix (note that ⁄
i ≠ ⁄
j = ⁄ i ≠ max( ⁄ j , ⁄
k +1 )
since ⁄ j = 0 for j Ø k + 1 ).
While we do not know if Theorem 2.2 is tight for all choices of ⁄ and k , it does give atight bound for some problems.
Namely, when applied to the covariance matrix estimationproblem, in the special case where k =
d Theorem 2.2 implies a bound of E
[ Î ˆ M ≠ M Î F ] Æ˜ O ( Ô kd )
=
O ( d ) (see Corollary 2.3).
Since ˆ M ≠ M = Ô 2log( 1 . 25 ” ) Á ( G + G € ) , the matrixˆ M ≠ M has independent Gaussian entries with mean zero and variance ˜ O
(1) , and we havefrom concentration results for Gaussian random matrices (see e.g. Theorem 2.3.6 of [40])that E [ Î ˆ M ≠ M Î F ] = ˜
( d ) , implying that the bound in Theorem 2.2 is tight in this case.
The proof of Theorem 2.2 di  ers from prior works, including that of [19] which use DavisKahan-type theorems [13] and trace inequalities, and instead relies on an interpretation ofthe Gaussian mechanism as a di  usion process which may be of independent interest.
Thisconnection allows us to use sophisticated tools from stochastic di  erential equations andrandom matrix theory.
We present an outline of the proof in Section 4.
Application to covariance matrix approximation: Plugging ⁄
i =
i for i Æ k and 194 ⁄
i
=
for i > k into Theorem 2.2, and plugging in concentration bounds for the perturbationto the eigenvalues ‡ i , we obtain utility bounds for covariance matrix approximation:
Corollary 2.3 ( Rank- k covariance matrix approximation). Let Á , ” > 0 , and givena symmetric matrix M œ R d ◊ d with eigenvalues ‡ 1 Ø · · · Ø ‡ d and corresponding orthonormal eigenvectors v 1 , . .
. , v d .
Let G be a matrix with iid N (0 , 1) entries, and considerthe mechanism that outputs ˆ M = M + Ô 2log( 1 . 25 ” )
Á ( G + G € ) .
Then such a mechanism is( Á , ” ) -di
erentially private.
Moreover, for any k œ [ d ] , deﬁne  k :=
diag( ‡ 1 , . . .
, ‡ k , 0 . . .
, 0)and V =
[ v 1 , . . .
, v d ] , and deﬁne ˆ ‡ 1 Ø · · · Ø ˆ ‡ d to be the eigenvalues of ˆ M with corresponding orthonormal eigenvectors ˆ v 1 , . .
, ˆ v d , and deﬁne ˆ  k :=
diag(ˆ ‡ 1 , . .
, ˆ ‡ k , 0 . .
, 0)and ˆ V :=
[ˆ v 1 , . . .
, ˆ v d ] .
Then if M satisﬁes Assumption 2.1 for ( M, k, ‡ 1 , Á , ” ), and deﬁning‡ d +1 := 0 , we have E [Equation]
[Equation]
The proof appears in Appendix G.
If ‡ k ≠ ‡ k +1 =
( ‡ k ) , then Corollary 2.3 implies that
E Ë Î
[Equation]
Thus, for matrices M with eigenvalues satisfying Assumption 2.1 and where ‡ k ≠ ‡ k +1 =
( ‡ k ) , Corollary 2.3 improves by a factor ofÔ k on the bound in Theorem 7 of [19] which says Î ˆ V ˆ
k
ˆ V €
≠ M Î F ≠ Î V  k V € ≠ M Î F =˜
O ( k Ô d ) w.h.p..
This is because an upper bound on Î ˆ V ˆ
k
ˆ V € ≠ V  k V € Î F implies an upperbound on Î ˆ V ˆ
k
ˆ V €
≠ M Î F ≠ Î V  k V € ≠ M Î F by the triangle inequality.
On the other 211 hand, while their result does not require a bound on the gaps in the eigenvalue of M andbounds their utility w.h.p.
, our Corollary 2.4 requires a bound on the gaps of the top k + 1eigenvalues of M and bounds the expected utility E [ Î ˆ V ˆ
k
ˆ V € ≠ V  k V € Î F ] .
Application to subspace recovery: Plugging in ⁄ 1 = · · · = ⁄ k
= 1 and ⁄ k +1 = · · · = 215 ⁄ d = 0 , the post-processing step in Theorem 2.2 outputs a projection matrix, and we obtainutility bounds for the subspace recovery problem.
Corollary 2.4 ( Subspace recovery). Let Á , ” > 0 , and given a symmetric matrix M œ
R d ◊ d with eigenvalues ‡ 1 Ø · · · Ø ‡ d and corresponding orthonormal eigenvectors v 1 , . .
. , v d .
Let G be a matrix with iid N (0 , 1) entries, and consider the mechanism that outputsˆ M = M + Ô 2log( 1 . 25 ” )
Á ( G + G € ) .
Then such a mechanism is ( Á , ” ) -di
erentially private.
Moreover, for any k œ [ d ] , deﬁne the d ◊ k matrices V k
= [ v 1 , . .
, v k ] and ˆ V k
= [ˆ v 1 , . . .
, ˆ v k ] ,where ˆ ‡ 1 Ø · · · Ø ˆ ‡ d denote the eigenvalues of ˆ M with corresponding orthonormaleigenvectors ˆ v 1 , . . .
, ˆ v d .
Then if M satisﬁes Assumption 2.1 for ( M, k, 2 , Á , ” ), we have
E Ë Î
[Equation] [Equation]
Moreover, if we also have that ‡ i ≠ ‡ i +1 Ø
( ‡ k ≠ ‡ k +1 ) for all
i Æ k , then E
[Equation]
The proof appears in Appendix H. For matrices M satisfying Assumption 2.1, the ﬁrstinequality of Corollary 2.4 recovers (in expectation) the Frobenius-norm utility bound implied by Theorem 6 of [19], which states that Î ˆ V
k ˆ V € k ≠ V k V € k Î F
[Equation]
w.h.p.
Moreover, for many input matrices M with spectral proﬁles ‡ 1 Ø · · · Ø ‡ d satisfying 230 Assumption 2.1, Theorem 2.2 implies stronger bounds than those in [19] for the subspacerecovery problem.
For instance, if we also have that ‡ i ≠ ‡ i +1 Ø  ( ‡ k ≠ ‡ k +1 ) for all i Æ k , 232 the bound given in the second inequality of Corollary 2.4 improves on the bound of [19] bya factor of Ô k .
On the other hand, while their result only requires that ‡ k
≠ ‡ k +1 Ø Ô dand bounds the Frobenius distance Î ˆ V ˆ V ≠ V
V Î F w h our Corollary 4 requires a on the of the 1 eigenvalues
M bounds expected Frobenius
ˆ
ˆ
Preliminaries
Brownian motion and stochastic calculus.
A Brownian motion W ( t ) in R is a continuous process that has stationary independent increments (see e.g., [34]).
In a multidimensional Brownian motion, each coordinate is an independent and identical Brownianmotion.
The ﬁltration F t generated by W ( t ) is deﬁned as ‡ ( ﬁ s Æ t
‡ ( W ( s ))) , where ‡ (  ) isthe ‡ -algebra generated by  .
W ( t ) is a martingale with respect to F t .
243 Deﬁnition 3.1 ( Itô Integral).
Let W ( t ) be a Brownian motion for t Ø 0 , let F t be the 244 ﬁltration generated by W ( t ) , and let z ( t ) :
F t æ R be a stochastic process adapted to F t .
The Itô integral is deﬁned as
[Equation]
Lemma 3.1 ( Itô’s Lemma, special case; Theorem 3.7.2 of [31]). Let f : R n æ R beany twice-di  erentiable function.
Let W ( t ) œ R n , be a Brownian motion.
Then d f ( W ( t )) =
Ò f ( W ( t )) € d W ( t ) + 12  f ( W ( t ))
d t, where  f :=
[Equation]
i is the Laplacian.
Lemma 3.2 ( Itô Isometry (one-dimension); Lemma 3.1.5 of [35]). Let W ( t ) be a
Brownian motion for t Ø 0 , let F t be the ﬁltration generated by W ( t ) , and z ( t ) :
F t æ R astochastic process adapted to F t .
Then we have E
[Equation]
Ës
[-T 
-]
[Equation]
Corollary 3.3 ( Itô Isometry (matrix-valued stochastic processes)). Let W ( t )be a d ◊ d -dimensional Brownian motion for t Ø 0 , let F t be the ﬁltration generated 254 by W ( t ) , and let Z ( t ) :
F t æ R d ◊ d be a stochastic process adapted to F t .
Then
E 5..
[Equation]
, where the Hadamard product § denotesentry-wise multiplication
Dyson Brownian motion.
Let W ( t ) œ
R
d
◊ d be a matrix where each entry is anindependent standard Brownian motion with distribution N (0 , tI d ) at time t , and let
B ( t ) =
W ( t ) + W € ( t ) .
Deﬁne the symmetric-matrix valued stochastic process  ( t ) asfollows:
[Equation]
The process  ( t ) is referred to as (matrix) Dyson Brownian motion.
At every time t > 0the eigenvalues “ 1 ( t ) , . . .
, “ d ( t ) of  ( t ) are distinct with probability 1 , and (1) induces astochastic process on the eigenvalues and eigenvectors.
The process on the eigenvalues andeigenvectors can be expressed via the following di  usion equations.
The eigenvalue di  usionprocess, which is also referred to as (eigenvalue) “Dyson Brownian motion”, is deﬁned bythe stochastic di  erential equation (2).
The (eigenvalue) Dyson Brownian motion is an
Itô di  usion and can be expressed can be expressed by the following stochastic di  erentialequation [20]:
[Equation]
The corresponding eigenvector process v 1 ( t ) , . .
. , v d ( t ) , referred to as the Dyson vector ﬂow,is also an Itô di  usion and, conditional on the eigenvalue process (2), can be expressed bythe following stochastic di  erential equation (see e.g., [3]):
[Equation]
Eigenvalue bounds.
The following two Lemmas will help us bound the gaps in theeigenvalues of the Dyson Brownian motion:
Lemma 3.4 (Theorem 4.4.5 of [44], special case 1 ) . Let W œ R
d ◊ d with i.i.d.
N (0 , 1)entries.
Then P ( Î W Î 2 > 2( Ô d + s ) < 2 e ≠ s 2 for any s > 0 .
Lemma 3.5 ( Weyl’s Inequality; [6]).
If A, B œ R d ◊ d are two symmetric matrices,and denoting the i ’th-largest eigenvalue of any symmetric matrix M by ‡ i ( M ) , we have‡ i ( A ) + ‡ d ( B ) Æ
‡ i ( A + B ) Æ ‡ i ( A )
+ ‡ 1 ( B ) .
Proof of Theorem 2.2
We give an overview of the proof of Theorem 2.2, along with the main technical lemmas usedto prove this result.
Section 4.1 outlines the di  erent steps in our proof.
In Steps 1 and 2 weconstruct the matrix-valued di  usion used in our proof.
Steps 3,4, and 5 present the maintechnical lemmas, and in step 6 we explain how to complete the proof.
The statements ofthe lemmas and the highlights of their proofs, are given in Sections 4.2, 4.3, 4.4.
In section4.5 we explain how to complete the proof.
The full proofs are deferred to the appendix.
4. Outline of proof
Step 1: Expressing the Gaussian Mechanism as a Dyson Brownian Motion.
Toobtain our utility bound, we view the Gaussian mechanism as a matrix-valued Brownianmotion (1) initialized at the input matrix M :  ( t ) :
=
M + B ( t ) ’
t Ø 0 .
If we run this
Brownian motion for time T = Ô 2log( 1 . 25 ” ) / Á we have that  ( T ) =
( Ô 2log( 1 . 25 ” ) / Á )( G + G € ) ,recovering the output of the Gaussian mechanism.
In other words, the input to the
Gaussian mechanism is M =  (0) , and the output is ˆ M =  ( T ) .2.
Step 2: Expressing the post-processed mechanism as a matrix di  usion  ( t ) .
Our goal is to bound Î ˆ V
ˆ V € ≠ V  V € Î F , where M = V  V € and ˆ
M =
ˆ V ˆ
ˆ V € arespectral decompositions of M and ˆ M .
To bound the error Î ˆ V
ˆ V € ≠ V  V € Î F we willdeﬁne a stochastic process  ( t ) such that  (0) =
V  V € and  ( t ) =
ˆ V  ˆ V € , and thenbound the Frobenius distance Î  ( T ) ≠  (0) Î F by integrating the (stochastic) derivativeof  ( t ) over the time interval [0 , T ] .
Towards this end, at every time t , let  ( t ) =
U ( t )  ( t ) U ( t ) € be a spectral decompositionof the symmetric matrix  ( t ) , where  ( t ) is a diagonal matrix with diagonal entries“ 1 ( t ) Ø · · · Ø “ d ( t ) that are the eigenvalues of  ( t ) , and U ( t ) =
[ u 1 ( t ) , · · · , u d ( t )] is a d ◊ dorthogonal matrix whose columns u 1 ( t ) , · · · , u d ( t ) are an orthonormal basis of eigenvectorsof  ( t ) .
At every time t , deﬁne  ( t ) to be the symmetric matrix with eigenvalues  andeigenvectors given by the columns of U ( t ) :
( t ) :
= U ( t )  U ( t ) € ’ t œ [0 , T ] .3.
Step 3:
Computing the stochastic derivative d  ( t ) .
To bound the expected squared
Frobenius distance E [ Î  ( T )
≠  (0) Î 2 F ] , we ﬁrst compute the stochastic derivative d  ( t )of the matrix di  usion  ( T ) (Lemma 4.2).4.
Step 4:
Bounding the eigenvalue gaps.
The equation for the derivative d  ( t ) includes terms with magnitude proportional to the inverse of the eigenvalue gaps  ij ( t ) :
=“ i ( t ) ≠ “ j ( t ) for each i, j œ [ d ] , which evolve over time.
In order to bound these terms,we use Weyl’s inequality (Lemma 3.5) to show that w.h.p.
the gaps in the top k + 1eigenvalues
ij ( t ) satisfy  ij ( t ) Ø  ( ‡ i ≠ ‡ j ) for every time t œ [0 , T ] , provided that theinitial gaps are su  ciently large (Lemma 4.4).5.
Step 5: Integrating the stochastic di  erential equation.
Next, we express theexpected squared Frobenius distance E
[ Î  ( T ) ≠  (0) Î 2 F ] as an integral Î  ( T ) ≠  (0) Î 2 F ] =
E 5..
.s
T
d  ( t ) ...
F. We then apply the Itô Isometry for matrices (Corollary 3.3) to obtaina formula for this integral.
Roughly speaking, the formula we obtain (Lemma 4.5) is
[Equation]
Step 6: Completing the proof.
Plugging the bound  ij ( t ) Ø  ( ‡ i ≠ ‡ j ) into (4), andnoting that the ﬁrst term on the RHS of (4) is at least as large as the second term since‡ i ≠ ‡
j Ø Ô d , we obtain the bound in Theorem 2.2.
4. Step 3: Computing
the
stochastic derivative d
( t )
( t ) is itself a matrix-valued di  usion.
We use the eigenvalue and eigenvector dynamics2 and 3 together with Itô’s Lemma (Lemma 3.1) to compute the Itô derivative of thisdi  usion.
Towards this end, we ﬁrst decompose the matrix  ( t ) as a sum of its eigenvectors: ( t ) =
q di =
⁄
i u
i ( t ) u € i ( t ) .
Thus, we have
[Equation] [Equation]
We begin by computing the stochastic derivative d( u i ( t ) u € i ( t )) for each i œ [ d ] , by applyingthe formula for the derivative of u i ( t ) in (3), together with Ito’s Lemma (Lemma 3.1):
Lemma 4.1 ( Stochastic derivative of u i ( t ) u € j ( t ) ).
For all t œ [0 , T ] , d( u i ( t ) u € i ( t )) =
[Equation]
The proof is in Appendix A. Plugging Lemma 4.1 into (5), we get an expression for d  ( t ) :
Lemma 4.2 ( Stochastic derivative of  ( t ) ; see Appendix B for proof).
For allt œ [0 , T ] we have that d  ( t ) =
[Equation]
4. Step 4: Bounding the eigenvalue gaps
The derivative in Lemma 4.2 contains terms with magnitude proportional to the inverse ofthe eigenvalue gaps  ij ( t ) :
= “ i ( t ) ≠ “ j ( t ) .
To bound these terms, we would like to showthat inf t œ [0 ,T ]  ij ( t )
Ø  ( ‡ i ≠ ‡ j ) for each
i < j Æ k +1 with high probability.
Towards this 338 end, we ﬁrst apply the spectral norm concentration bound for Gaussian random matrices(Lemma 3.4), which provides a high-probability bound for Î B ( t ) Î 2 at any time t , togetherwith Doob’s submartingale inequality, to show that the spectral norm of the matrix-valued
Brownian motion B ( t ) does not exceed T Ô d at any time t œ [0 , T ] w.h.p.:
Lemma 4. ( Spectral norm bound). For every T > 0 , we have,
P
[Equation]
The proof appears in Appendix C. Next, we use Lemma 4.3 to bound the eigenvalue gaps:
Lemma 4. ( Eigenvalue gap bound).
Whenever “ i (0) ≠ “ i +1 (0) Ø 4 T Ô dfor every i œ S and T > 0 and some subset S µ [ d ≠ 1] , we have
[Equation]
To prove Lemma 4.4, we plug Lemma 4.3 into Weyl’s Inequality (Lemma 3.5), to show that“ [Equation] ≠ ‡ [Equation]
+
[Equation]
) , with high probability for each i Æ k (Lemma 4.4).
The last inequality holds since Assumption 2.1 ensures ‡ i ≠ ‡ i +1 Ø 12 T Ô d for i
Æ k .
The full proof is in Appendix D.
4. Step 5: Integrating the stochastic di  erential equation
Next, we would like to integrate the derivative d  ( t ) to obtain an expression for E [ Î  ( T ) ≠ (0) Î 2 F ] , and to then plug in our high-probability bounds (Lemma 4.4) for the gaps  ij ( t ) .
To allow us to later plug in these high-probability bounds after we integrate and take theexpectation, we deﬁne a new di  usion process Z ÷ ( t ) which has nearly the same stochasticdi  erential equation as 4.2, except that each eigenvalue gap  ij ( t ) is not permitted tobecome smaller than the value ÷ ij = 14 ( ‡ i ≠ max( ‡ j , ‡ k +1 )) for each i < j .
Towards this end, ﬁx any ÷ œ R d ◊ d , deﬁne the following matrix-valued Itô di  usion Z ÷ ( t )via its Itô derivative d Z ÷ ( t ) :
[Equation]
with initial condition Z ÷ (0) :
=
(0) .
Thus, Z ÷ ( t ) =
(0) + s t
d Z ÷ ( s ) for all t Ø 0 .
We thenintegrate the d Z ÷ ( t ) over the time interval [0 , T ] , and apply the Itô isometry (Corollary 3.3)to obtain an expression for the Frobenius norm of this integral:
Lemma 4.5 ( Frobenius distance integral).
For any T > 0 , E
[Equation]
T[-
-]
[Equation]
[-T 
-]s
[Equation] [Equation]
[Equation]
To prove Lemma 4.5, we write E [ Î Z ( T ) ≠ Z (0) Î 2 F ] =12 E
[Equation] [Equation]
To compute the ﬁrst term on the RHS, we apply the Itô isometry (Corollary 3.3).
Thekey step is to notice that, since iid Gaussian matrices are invariant (in distribution) toorthogonal transformations, we have that U ( t )d B ( t ) U € ( t ) has the same distribution asd B ( t ) .
Moreover, since u i ( t ) , u j ( t ) ,  ij ( t ) are all di  usion processes w.r.t.
the ﬁltration F t generated by the Brownian motion B ( t ) , d B ( t ) is independent of the value ofthe eigenvectors u i ( · ) , u j ( · ) and eigenvalue gaps  ij ( · ) at all times · Æ t up to thepresent time t .
Thus, we can replace the term Ï ( t ) § d B ( t ) inside the integral, whereÏ ( t ) :
=
q
[Equation]
u
[Equation]
( t ) u € j ( t ) + u j ( t ) u € i ( t )) , with U ( t ) Ï ( t ) U € ( t ) § d B ( t ) =
[Equation]
2 § d B ( t ) , where e 1 , . . .
, e d are the standard basis vectors.
This allows us to apply the Itô isometry to each entry of the matrix individually to compute the ﬁrst term on the RHS.
To compute the second term on the RHS, we use the
Cauchy-Schwarz inequality.
The full proof appears in Appendix E.
4. Step 6: Completing the proof
To complete the proof, we plug in the high-probability bounds on the eigenvalue gaps from
Section 4.3 into Lemma 4.5.
Since by Lemma 4.4  ij ( t ) Ø 12 ( ‡ i ≠ ‡ j ) w.h.p.
for eachi, j Æ k + 1 , and ÷ ij = 14 ( ‡ i ≠ max( ‡ j , ‡ k +1 )) , we must also have that Z ÷ ( t ) =
( t ) forall t œ [0 , T ] w.h.p.
Plugging in the high-probability bounds  ij ( t ) Ø 12 ( ‡ i ≠ ‡ j ) for each 382 i, j Ø k + 1 , and noting that ⁄ i ≠ ⁄ j = 0 for all i, j > k , we get that [Equation] Ëq
[Equation]
q
[Equation]
q
[Equation]
q
[Equation] [Equation]
Since ( ‡ i ≠ max( ‡ j , ‡ k +1 )
Ø  ( Ô d ) for all i Æ k and j œ [ d ] , we can use the CauchySchwarz inequality to show that the second term is (up to a factor of T) smaller than theﬁrst term:
[Equation]
[Equation]
into (8), we obtain the bound in Theorem 2.2.
For the full proof of Theorem2.2, see Appendix F
Conclusion and Future Work
We present a new analysis of the Gaussian mechanism for a large class of symmetric matrixapproximation problems, by viewing this mechanism as a Dyson Brownian motion initializedat the input matrix M .
This viewpoint allows us to leverage the stochastic di  erentialequations which govern the evolution of the eigenvalues and eigenvectors of Dyson Brownianmotion to obtain new utility bounds for the Gaussian mechanism.
To obtain our utilitybounds, we show that the gaps in the eigenvalues of the Dyson Brownian motion stay atleast as large as the initial gap sizes (up to a constant factor), as long as the initial gaps inthe top k +1 eigenvalues of the input matrix are Ø  ( Ô d ) .
The main question that remainsopen is whether for other initial matrices M the gaps in the eigenvalues of Dyson Brownianmotion grow over time as well, and, if true, whether one can use this fact to obtain similarutility bounds for input matrices M with eigenvalue gaps that are smaller than Ô d .
Finally, this paper analyzes a mechanism in di  erential privacy, which has many implications for
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO]to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to youranswer , either by referencing the appropriate section of your paper or providing a briefinline description.
For example:
• Did you include the license to the code and datasets?
[N/A]• Did you include the license to the code and datasets? [N/A]• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Notethat the Checklist section does not count towards the page limit.
In your paper, please deletethis instructions block and only keep the Checklist section heading above along with thequestions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect
the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (See Section 5)
(d) Have you read the ethics review guidelines and ensured that your paper con- forms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (See Section 2 and Assumption 2.1)
(b) Did you include complete proofs of all theoretical results?
[Yes] (See Section4 and Appendix A, B, C, D, E,F, G,H)
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (See Appendix I)
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how
they were chosen)?
[Yes] (See Appendix I)
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?
[Yes] (See Appendix I)
(d) Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] (See Appendix I)
If you are using existing assets (e.g., code, data, models) or curating/releasing new
assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a
URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally
identiﬁable information or o  ensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screen-
shots, if applicable?
[N/A]
[Table] ) Did you describe any potential participant risks, with [Table] to Institutional
[Table]
Review Board (IRB) approvals, if applicable?
[N/A
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total
[Table]
amount spent on participant compensation?
[N/A
[Table]