Brownian Noise Reduction: Maximizing Privacy Subject to Accuracy Constraints
Abstract
There is a disconnect between how researchers and practitioners handle privacyutility tradeoffs.
Researchers primarily operate from a privacy ﬁrst perspective, setting strict privacy requirements and minimizing risk subject to these constraints.
Practitioners often desire an accuracy ﬁrst perspective, possibly satisﬁed with the greatest privacy they can get subject to obtaining sufﬁciently small error.
Ligett et al.
[2017] have introduced a “noise reduction” algorithm to address the latter perspective.
The authors show that by adding correlated Laplace noise and progressively reducing it on demand, it is possible to produce a sequence of increasingly accurate estimates of a private parameter while only paying a privacy cost for the least noisy iterate released.
In this work, we generalize noise reduction to the setting of Gaussian noise, introducing the Brownian mechanism .
The Brownian mechanism works by ﬁrst adding Gaussian noise of high variance corresponding to the ﬁnal point of a simulated Brownian motion.
Then, at the practitioner’s discretion, noise is gradually decreased by tracing back along the Brownian path to an earlier time.
Our mechanism is more naturally applicable to the common setting of bounded (cid:96) 2 -sensitivity, empirically outperforms existing work on common statistical tasks, and provides customizable control of privacy loss over the entire interaction with the practitioner.
We complement our Brownian mechanism with
ReducedAboveThreshold , a generalization of the classical AboveThreshold algorithm that provides adaptive privacy guarantees.
Overall, our results demonstrate that one can meet utility constraints while still maintaining strong levels of privacy.
Introduction
Over the past decade, differential privacy has seen industry-wide adoption as a means of protecting sensitive information [Erlingsson et al., 2014, Greenberg, 2016].
By injecting appropriate amounts of noise, differentially private algorithms allow the computation of population-level quantities of interest while guaranteeing individual-level privacy.
Of the private mechanisms used in industry, those relating to private empirical risk minimization (ERM) are perhaps the most impactful, in part due to their application in machine learning tasks [Abadi et al., 2016, Song et al., 2013].
Researchers have developed many private ERM mechanisms, ranging from least squares minimzation [Sheffet, 2017, Chaudhuri et al., 2011] to subsampled gradient descent [Abadi et al., 2016, Balle and Wang, 2018, Wang et al., 2019].
Despite this vast literature, most existing results take the same broad approach: they aim to minimize error (statistical risk) subject to strict privacy guarantees.
While this strict adherence to privacy constraints may be necessary in some applications, it often provides weak utility guarantees [Fienberg et al., 2010] and can make some learning tasks impossible
[Dwork et al., 2009].
Industry applications of differential privacy may desire an accuracy ﬁrst perspective, setting desired risk requirements for models used in production.
Privacy may still be a desirable aspect of computation, but it is by no means the only goal; minimizing risk may take center stage.
The main existing approach to this accuracy-oriented perspective on privacy was given by Ligett et al.
These authors introduce a noise reduction mechanism for gradually releasing a private, high-dimensional parameter.
By leveraging a Laplace-based Markov process [Koufogiannis et al., 2017], they construct a mechanism for which the privacy loss of releasing arbitrarily many estimates of a parameter only depends on the privacy loss of the least noisy parameter viewed.
This is in contrast to results about the composition of private algorithms, in which privacy degrades according to the total number of parameters witnessed [Dwork et al., 2010, Kairouz et al., 2015, Murtagh and Vadhan, 2016].
The authors also demonstrate how to privately query the utility of observed parameters on private data by coupling their Laplace-based mechanism with AboveThreshold , a classical differentially private algorithm
[Dwork and Roth, 2014, Lyu et al., 2017].
While the above mechanism provides signiﬁcant privacy loss savings over a baseline method that doubles the privacy loss each round, Laplace noise is unﬁt for many settings in which (cid:96) 2 -sensitivity is used for calibrating noise.
Since converting from (cid:96) 2 -sensitivity to (cid:96) 1 -sensitivity 1 incurs a dimensiondependent cost, it is important to develop a noise reduction technique with Gaussian noise.
Contributions and paper outline.
We introduce the Brownian mechanism , a novel approach for privately releasing a parameter vector subject to accuracy constraints.
The Brownian mechanism adds correlated Gaussian noise to a risk-minimizing parameter through a Brownian motion.
Noise is then iteratively stripped by moving adaptively backwards along the random walk until a suitable stopping condition is met, such as meeting a target accuracy on a public dataset.
In Section 3, we deﬁne the
Brownian mechanism and characterize its privacy loss.
Using machinery from martingale theory, we construct privacy boundaries for the Brownian mechanism — upper bounds on privacy loss that hold simultaneously with high probability.
In particular, the failure probability of these bounds does not depend on the number of outcomes observed, overcoming a seeming need for a union bound faced by Ligett et al.
These privacy boundaries yield provable, high-probability bounds on privacy loss under data-dependent stopping conditions.
If private data is used to evaluate risk, then the data-dependent stopping conditions can themselves leak information.
To counter this, we introduce ReducedAboveThreshold in Section 5, a generalization of the classical AboveThreshold algorithm for privately querying accuracy on sensitive data.
We show how to couple ReducedAboveThreshold and the Brownian mechanism so that a data analyst only ever incurs twice the privacy loss they would incur if they had queried accuracy on a public dataset.
This is in contrast to the results in Ligett et al.
[2017], which note that the privacy loss of
AboveThreshold often dominates the privacy loss incurred from using noise reduction.
We empirically evaluate the Brownian mechanism and ReducedAboveThreshold in Section 6, ﬁnding that the Brownian mechanism can offer privacy loss savings over the Laplace noise reduction method introduced by Ligett et al.
In our view, these results demonstrate that the Brownian mechanism is a practical, intuitive mechanism for meeting accuracy requirements in private ERM.
Lastly, we derive other new mechanisms for noise reduction, of independent interest.
We generalize the Laplace process of Koufogiannis et al.
[2017] to continuous time in Section 4, thus making the Laplace noise reduction mechanism of Ligett et al.
[2017] more ﬂexible and adaptive to datadependent privacy levels.
We also brieﬂy mention a noise reduction mechanism for Skellam noise in
Appendix E, a discrete distribution used in count queries [Agarwal et al., 2021].
Preliminaries
Differential privacy, privacy loss, and ex-post privacy.
An algorithm A : X → Y is ( (cid:15), δ ) - differentially private if, for any measurable set E ⊂ Y and any neighboring inputs x ∼ x (cid:48) ,
[Equation]
In the above [Dwork et al., 2006], ∼ denotes some arbitrary neighboring relation.
Typically x ∼ x (cid:48) indicates x and x (cid:48) differ in one entry, but any other relation sufﬁces.
While differential privacy has proven itself a mainstay of private computation, condition (1) is too rigid to allow data analysts to achieve a minimum desired accuracy.
In other words, it embraces a privacy ﬁrst perspective, ﬁxing a strict condition in terms of parameters (cid:15) and δ that must be met.
We are interested in the accuracy ﬁrst perspective, setting a target accuracy and correspondingly optimizing privacy parameters.
The above deﬁnition of differential privacy is qualitatively focused on bounding the informationtheoretic quantity of privacy loss [Dwork et al., 2006, 2010, Dwork and Roth, 2014].
Deﬁnition 2.1 ( Privacy Loss ) .
Let A : X → Y be an algorithm, and ﬁx neighbors x ∼ x (cid:48) in X .
Let p x and p x (cid:48) be the respective densities of A ( x ) and A ( x (cid:48) ) on the space Y with respect to some reference measure 2 .
Then, the privacy loss between A ( x ) and A ( x (cid:48) ) is the random variable
[Equation]
We think of A ( x ) as the true outcome, and L ( x, x (cid:48) ) measures how much more likely this outcome is under the true input x versus an alternative x (cid:48) .
Privacy loss provides a probabilistic deﬁnition of privacy.
Namely, A is ( (cid:15), δ ) -probabilistically differentially private if, for all neighbors x ∼ x (cid:48) ,
[Equation]
While probabilistic differential privacy is not equivalent to differential privacy [Kasiviswanathan and Smith, 2014], ( (cid:15), δ ) -probabilistically differential privacy implies ( (cid:15), δ ) -differential privacy.
Probabilistic differential privacy emerged as a means for studying privacy composition, and has been leveraged in proving many results [Kairouz et al., 2015, Murtagh and Vadhan, 2016, Rogers et al., 2016, Whitehouse et al., 2022].
A natural extension of privacy to the accuracy-oriented regime is ex-post privacy, which allows the bound in condition (2) to depend the observed algorithm output.
Deﬁnition 2.2 (Ligett et al.
Let A : X → Y be an algorithm and E : Y → R ≥ 0 a function.
We say A is ( E , δ ) -ex-post private if, for any neighboring inputs x ∼ x (cid:48) , we have
[Equation]
While any algorithm is trivially ex-post private with E ( A ( x )) := ∞ , the goal is to make E ( A ( x )) as small as possible.
We describe theoretical tools for obtaining ex-post privacy guarantees in Section 3, and empirically compute the ex-post privacy distributions of various mechanisms in Section 6.
Background on Noise Reduction.
Heuristically, a noise reduction mechanism allows a data analyst to view multiple, increasingly accurate estimates of a risk minimizing parameter while only paying an ex-post privacy cost for the least noisy iterate observed.
More formally, let M : X → Y ∞ be any algorithm mapping databases to sequences of outputs.
The indices of the sequence denote the rounds of interaction, with smaller indices indicating greater noise.
Let M n : X → Y give the n th element of the sequence and M 1: n : X → Y n the ﬁrst n elements.
Deﬁnition 2.3 ( Noise Reduction Mechanism ) .
We say M : X → Y ∞ is a noise reduction mechanism if, for any n ≥ 1 and any neighboring datasets x ∼ x (cid:48) , we have
[Equation]
where L 1: n ( x, x (cid:48) ) denotes the privacy loss between M 1: n ( x ) and M 1: n ( x (cid:48) ) and L n ( x, x (cid:48) ) is the privacy loss between M n ( x ) and M n ( x (cid:48) ) .
The only noise reduction mechanism in the literature uses a Markov process with Laplace marginals [Koufogiannis et al., 2017] to gradually release a sensitive parameter [Ligett et al., 2017].
As originally presented, this Laplace Noise Reduction mechanism is nonadaptive, requiring a data analyst to ﬁx a ﬁnite sequence of privacy parameters ( (cid:15) n ) n ∈ [ K ] in advance.
Instead of presenting this method as background, we describe it in Section 4, in which we construct an adaptive generalization of this mechanism.
We then leverage this generalization as a subroutine in ReducedAboveThreshold , a generalization of AboveThreshold with adaptive privacy guarantees.
Background on Brownian Motion.
We now provide a brief background on Brownian motion, perhaps the best-known example of a continuous time stochastic process [Le Gall, 2016].
Deﬁnition 2.4.
A continuous time real-valued process ( B t ) t ≥ 0 is called a standard Brownian motion if (1) B 0 = 0 , (2) ( B t ) t ≥ 0 has continuous sample paths, (3) ( B t ) has independent increments, i.e.
B t + s − B s is independent of B s for all s, t ≥ 0 , and (4) B t ∼ N (0 , t ) for all t ≥ 0 .
We say a process ( B t ) t ≥ 0 is a d -dimensional standard Brownian motion if each coordinate process is an independent standard Brownian motion.
We use many properties of Brownian motion to construct the Brownian mechanism and analyze its privacy loss in Section 3.
One important property of Brownian motion is that it is a continuous time martingale.
This property allow us to use time-uniform supermartingale concentration to characterize and bound the privacy loss of the Brownian mechanism at data-dependent stopping times [Howard et al., 2020, 2021].
We do not go into detail about martingale concentration in this background section, but rather defer it to Appendix A. Additionally, ( B t ) t ≥ 0 is a Markov process.
This tells us that if we inspect the Brownian motion at times 0 ≤ t 1 < t 2 < · · · < t n , then B t 2 , . . .
, B t n can be viewed as a randomized post-processing of B t 1 that does not depend on B s for any s < t 1 .
This property allows us to show that the privacy loss of the Brownian mechanism — which adds noise to a parameter via a Brownian motion — only depends on the least noisy parameter observed.
The Brownian Mechanism: a Gaussian Noise Reduction Mechanism
The Brownian mechanism works by simulating a Brownian motion starting at some multivariate parameter; this parameter should be thought of as the risk-minimizing output if there were no privacy constraints.
The data analyst ﬁrst observes the random walk at some large time.
Then, if so desired, the analyst “rewinds” time to an earlier point on the Brownian path, reducing noise to obtain a more accurate estimate.
Due to the Markovian nature of Brownian motion, the analyst will only pay a privacy cost proportional to variance of the random walk at the earliest inspected time.
In what follows, when we refer to a sequence ( T n ) n ≥ 1 of time functions , we mean a sequence of functions
T n : ( R d ) n − 1 → R ≥ 0 such that, for all n and β 1: n ∈ ( R d ) n , T n +1 ( β 1: n ) ≤ T n ( β 1: n − 1 ) .
Deﬁnition 3.1.
Let f : X → R d be a function and ( T n ) n ≥ 1 a sequence of time functions.
Let ( B t ) t ≥ 0 be a standard d -dimensional Brownian motion.
The Brownian mechanism associated with f and ( T n ) n ≥ 1 is the algorithm BM : X → ( R d ) ∞ given by
[Equation]
where we set T n ( x ) :
= T n (cid:0) f ( x
[Equation]
+ B T 1 ( x ) , . . . , f ( x ) + B T n − 1 ( x ) (cid:1) with T 1 ( x ) being constant.
We have chosen T n ( x ) as indexing notation to denote dependence on x , even if this is only through observed parameters.
In the context of ERM, one can think of f as computing a risk minimizing parameter associated with a private dataset x ∈ X .
The data analyst uses T n along with the ﬁrst n −iterates to determine how far to rewind time to obtain the n th iterate.
Due to the Markovian nature of
Brownian motion, we get the following lemma.
We include a proof in Appendix B for completeness.
Lemma 3.2.
Let x ∼ x (cid:48) be neighbors.
For any n ≥ 1 , let L BM1: n ( x, x (cid:48) )
denote the privacy loss between
BM 1: n ( x ) and BM 1: n ( x (cid:48) ) and L BM n ( x, x (cid:48) )
the privacy loss between BM n ( x ) and BM n ( x (cid:48) ) .
Then,
L BM1: n ( x, x (cid:48) )
=
L BM n ( x, x (cid:48) ) .
Lemma 3.2 just tells us that the Brownian mechanism is a noise reduction mechanism, i.e. that the privacy lost by viewing the ﬁrst n iterates is exactly the privacy lost by viewing the n th iterate in isolation.
Thus, we can identify L BM1: n ( x, x (cid:48) ) with L BM n ( x, x (cid:48) ) going forward.
The Brownian mechanism, as deﬁned above, produces an inﬁnite sequence of parameters.
In practice, a data analyst will only view ﬁnitely many iterates, stopping when some utility condition has been met or a minimum privacy level is reached.
We introduce stopping functions to model how a data analyst adaptively interacts with noise reduction mechanisms.
Deﬁnition 3.3 ( Stopping Function ) .
Let M : X → Y ∞ be a noise reduction mechanism.
For x ∈ X , let ( F n ( x )) n ∈ N be the ﬁltration given by F n ( x ) :
= σ ( M i ( x ) : i ≤ n ) .
3 A function
N : Y ∞ → N is called a stopping function if for any x ∈ X , N ( x ) :
= N ( M ( x )) is a stopping time with respect to ( F n ( x )) n ≥ 1 .
A stopping function
N is a rule used to decide when to stop viewing parameters that only depends on the observed iterates of the noise reduction mechanism.
N could heuristically be “stop at the ﬁrst time a parameter achieves an accuracy of 95% on a held-out dataset.”
If a data analyst uses a stopping function alongside BM , per Deﬁnition 2.3, the privacy loss accrued upon stopping is L BM N ( x ) ( x, x (cid:48) ) .
In some cases, an analyst may want the stopping function to depend on the underlying private dataset through more than just the released parameters, e.g. they may want their rule to be “stop at the ﬁrst time a parameter achieves an accuracy of 95% on the private dataset.”
In this case, additional privacy may be lost due to observing N ( x ) .
We detail how to handle this more subtle case in Section 5.
The following theorem characterizes the privacy loss of the Brownian mechanism.
Theorem 3.4.
Let BM be the Brownian mechanism associated with ( T n ) n ≥ 1 and a function f .
For neighbors x ∼ x (cid:48) , the privacy loss between BM 1: n ( x ) and BM 1: n ( x (cid:48) ) is given by
[Equation]
where ( W t ) t ≥ 0 is a standard, univariate Brownian motion.
Suppose f has (cid:96) 2 -sensitivity at most ∆ 2 .
Then, letting a + := max(0 , a ) , we have
[Equation]
Theorem 3.4 also holds when a deterministic time n is replaced by N ( x ) , where N is a stopping function.
The above theorem can be viewed as a process-level equivalent of the well-known fact that the privacy loss of the Gaussian mechanism has an uncentered Gaussian distribution [Balle and Wang, 2018].
We prove the Theorem 3.4 in Appendix B. Given the clean characterization of privacy loss above, we now show how to construct high-probability, time-uniform privacy loss bounds.
We deﬁne privacy boundaries , which map the variance of BM to high-probability bounds on privacy loss.
Deﬁnition 3.5.
A function ψ : R ≥
→ R ≥ 0 is a δ -privacy boundary for the Brownian mechanism associated with time functions ( T n ) n ≥ 1 if for any neighboring datasets x ∼ x (cid:48) , we have
[Equation]
Since the privacy loss of BM is a deterministic function of a Brownian motion, we can apply results from martingale theory to construct general families of privacy boundaries.
Theorem 3.6.
Assume the same setup as in Theorem 3.4.
Let δ > 0 and f be a function with (cid:96) 2 -sensitivity ∆ 2 .
The following classes of functions form δ -privacy boundaries.
(Mixture boundary)
For any ρ > 0 , ψ Mρ given by
[Equation]
(Linear boundary)
For any a, b > 0 such that 2 ab = log(1 /δ ) , ψ La,b given by
[Equation]
We prove Theorem 3.6 in Appendix B.
In the same appendix, we plot the boundaries in Figure 4.
Privacy boundaries serve a dual purpose for the Brownian mechanism.
First, since time-uniform concentration bounds are valid at stopping times [Howard et al., 2021], privacy boundaries provide ex-post privacy guarantees.
Second, in many settings, it may be more natural for a data analyst to adaptively specify target privacy levels instead of noise levels.
This is, for instance, the case in our experiments in Section 6.
By inverting privacy boundaries, data analysts can compute the proper amount of noise to remove at each step to meet target privacy levels.
We make the above precise in Corollary 3.7.
In what follows, when we refer to a sequence ( E n ) n ≥of privacy functions , we mean a sequence of functions E n : ( R d ) n − 1 → R ≥ 0 such that, for all n and β 1: n ∈ ( R d ) n , E n +1 ( β 1: n ) ≥ E n ( β 1: n − 1 ) .
Corollary 3.7.
Let N be a stopping function, as in Deﬁnition 3.3.
If ψ is a δ -privacy boundary for
BM , we have
[Equation]
ψ (cid:0) T N ( x ) ( x ) (cid:1)(cid:17) ≤ δ, i.e. the algorithm BM 1: N ( · ) ( · ) is (cid:0) ψ ( T N ( · ) ( · )) , δ (cid:1)
-ex post private, where ( · ) denotes a positional argument for an input x ∈ X .
Further, let ( E n ) n ≥ 1 be a sequence of privacy functions, and deﬁne
T n ( β 1: n − 1 ) : = inf { t ≥ 0 : ψ ( t ) ≥ E n ( β 1: n − 1 ) } .
Then BM 1: N ( · ) ( · ) is ( E N ( · ) ( · ) , δ ) -ex post private, where E n ( x ) is deﬁned analogously to T n ( x ) .
Again, N should be thought of as a stopping rule based on parameter accuracy.
E n should be thought of as a rule for choosing the n th privacy parameter given BM 1: n − 1 ( x ) .
An Adaptive, Continuous-Time Extension of Laplace Noise Reduction
Here, we generalize the original noise reduction mechanism of Ligett et al.
[2017], which will be used as a subroutine in Algorithm 1 in the following section.
We ﬁrst describe the original Laplace-based
Markov process of Koufogiannis et al.
Fix any positive integer K and any ﬁnite, increasing sequence of times ( t n ) n ∈
[ K ] .
Let ( ζ n )
Kn =0 be the d -dimensional process given by ζ 0 = 0 and ζ n = (cid:40) ζ n −with probability (cid:16) t n − 1 t n (cid:17)ζ n − 1 + Lap( t n ) otherwise.
Koufogiannis et al.
[2017] show that ζ n ∼ Lap( t n ) and that ( ζ n )
Kn =0 is Markovian.
Ligett et al. [2017] use the above process to construct a noise reduction mechanism.
Namely, they deﬁne the the Laplace Noise Reduction mechanism associated with f : X → R d and ( t n ) n ∈
[ K ] to be the algorithm LNR : X → ( R d ) K given by LNR( x ) :
= ( f ( x ) + ζ K , . . .
, f ( x ) + ζ 1 ) .
If t n := ∆ 1 /(cid:15) n , then releasing n th component LNR n ( x ) in isolation is equivalent to running the classical Laplace mechanism with privacy level (cid:15) n .
We now extend the process ( ζ n ) n ∈ [ K ] to a continuous time process with the same ﬁnite-dimensional distributions.
Let η > 0 be arbitrary, and let ( P t ) t ≥ η be an inhomogeneous Poisson process with intensity function λ ( t ) :
= 2 t .
For n ≥ 1 , let T n :=
inf { t ≥ η : P t ≥ n } be the n th jump of ( P t ) t ≥ η and set T 0 :
=
η .
Noting that P t must be a nonnegative integer, deﬁne the process ( Z t ) t ≥ η by Z t
[Equation]
It is immediate that ( Z t ) t ≥ η is Markovian.
We show in Appendix D that Z t ∼ Lap( t ) .
With ( Z t ) t ≥ η , one can make LNR fully adaptive, meaning that the times ( t n ) n ∈ [ K ] at which it is invoked need not be prespeciﬁed, and can depend on the underlying input database x by using time functions.
Deﬁnition 4.1.
Let f : X → R d be a function and ( T n ) n ≥ 1 a sequence of time functions.
Let ( Z t ) t ≥ η be the process deﬁned in Equation (4) .
The Laplace noise reduction mechanism associated with f and ( T n ) n ≥ 1 is the algorithm LNR : X → ( R d ) ∞ given by
[Equation]
where again T n ( x ) :
= T n ( f ( x ) + Z T 1 ( x ) , . . . ,
f ( x ) + Z
[Equation] x [Equation] If the analyst would prefer instead to specify privacy functions ( E n ) n ≥ 1 , they can do so by leveraging the corresponding time functions T n ( x ) :
=
∆ 1 / E n ( x ) , where E n ( x ) is deﬁned analogously to
T n ( x ) .
We leverage LNR in our experiments in Section 6 and the process ( Z t ) t ≥ 0 as a subroutine in constructing ReducedAboveThreshold .
LNR admits the following trivial ex-post privacy guarantee.
Proposition 4.2.
Let LNR be associated with ( T n ) n ≥ 1 and a function f , and suppose f has (cid:96) 1sensitivity ∆ 1 .
If N is stopping function, the algorithm LNR 1: N ( · ) ( · ) is (∆ 1 /T N ( · ) ( · ) , 0) -ex post private.
Privately Checking if Accuracy is Above a Threshold
In Section 3 we presented the Brownian mechanism, characterized its privacy loss, and showed how to obtain ex-post privacy guarantees for arbitrary stopping functions.
In particular, these stopping functions could be based on the accuracy of the observed iterates on public held-out data.
However, one may desire to privately check the accuracy of observed iterates on the dataset x ∈ X .
Ligett et al.
[2017] were able to accomplish this goal by coupling LNR with AboveThreshold , a classical algorithm for privately answering threshold queries [Dwork and Roth, 2014].
In the context of ERM, AboveThreshold iteratively checks if the empirical risk of each parameter is below a target threshold, stopping at the ﬁrst such occurrence.
The downside to AboveThreshold is that it requires a preﬁxed privacy level.
In empirical studies, Ligett et al.
[2017] found this ﬁxed privacy cost dominated the ex-post privacy guarantees, showing little beneﬁt to using noise reduction.
Below, we construct ReducedAboveThreshold , a generalization of AboveThreshold which provides ex-post privacy guarantees.
We show how to couple BM with ReducedAboveThreshold to obtain tighter ex-post privacy guarantees than coupling with AboveThreshold would permit.
In particular, if BM is run using parameters ( (cid:15) n ) n ≥ 1 and ReducedAboveThreshold indicates the N th parameter obtains sufﬁciently high accuracy, the privacy loss of the net procedure will be at most 2 (cid:15) N — only twice the privacy loss that would be accrued by testing on public data.
Require:
Algorithm Alg :
X → Y ∞ , parameter (cid:15) max > 0 , threshold τ , database x ∈ X , utility u : Y × X → R where u ( β, · ) is ∆ -sensitive ∀ β , privacy functions ( E n ) n ≥ 1 with E n ≤ (cid:15) max ∀ n .
for n ≥ 1 do (cid:15) n := E n (Alg 1: n − 1 ( x )) , T n := 2∆ /(cid:15) n ζ n := Z T n , where ( Z t ) t ≥ η in Eq.
(4) deﬁnes the LNR mechanism with η := 2∆ /(cid:15) max .
ξ n ∼ Lap (cid:16) 4∆ (cid:15) n (cid:17) if u (Alg n ( x ) , x ) + ξ n ≥ τ + ζ n then
Print 1 and HALT else
Print τ should be seen as a target accuracy, Alg as a mechanism for releasing a parameter (e.g. BM , LNR ), and u as evaluating the accuracy of Alg n ( x ) on x . (cid:15) max is an arbitrarily large maximum privacy level, used to prevent the user from examining ( Z t ) at arbitrarily small times.
The above generalizes to sequences of thresholds ( τ n ) n ≥ 1 and sequences ( u n ) n ≥ 1 of functions u n : Y n × X → R that are ∆ -sensitive in their second argument, but the added generality yields only marginal beneﬁts.
When
E n = (cid:15) for all n , Algorithm 1 recovers AboveThreshold as a special case.
The intuition behind
ReducedAboveThreshold is that by gradually removing Laplace noise from the threshold, a data analyst can ensure that privacy of the whole procedure only depends on the magnitude of Laplace noise added when the algorithm halts.
The following characterizes the privacy loss of Algorithm 1.
Theorem 5.1.
For any n ≥ 1 and neighboring datasets x ∼ x (cid:48) , let L Alg1: n ( x, x (cid:48) ) denote the privacy between Alg 1: n ( x ) and Alg 1: n ( x (cid:48) ) .
For any x ∈ X , deﬁne N ( x ) to be the ﬁrst round where
ReducedAboveThreshold run on input x ∈ X outputs 1, that is
[Equation]
Then, the privacy loss between ReducedAboveThreshold( x ) and ReducedAboveThreshold( x (cid:48) ) , denoted L RAT ( x, x (cid:48) ) , is bounded by
[Equation]
We prove Theorem 5.1 in Appendix C, where we also provide a utility guarantee for
ReducedAboveThreshold .
This utility guarantee, much like the utility guarantee for
AboveThreshold , is in practice weak as it derives from a union bound.
Using Theorem 5.1, we can simply choose Alg = BM as a means of adaptively generating parameters.
The following corollary, which follows immediately from the above theorem, provides the ex-post privacy guarantees of combining ReducedAboveThreshold and BM .
Corollary 5.2.
Let BM be the Brownian mechanism associated with a function f , decreasing time functions ( T n ) n ≥ 1 , and a a δ -privacy boundary ψ .
Let ReducedAboveThreshold be run with privacy functions ( ψ ( T n )) n ≥ 1 , threshold τ , and algorithm BM .
Then, ReducedAboveThreshold is (cid:0) 2 ψ ( T N ( · ) ( · )) , δ (cid:1)
-ex post private.
Experiments
[Figure]
Choice of tasks: We compare the performance of BM and LNR on the tasks of regularized logistic regression via output perturbation [Chaudhuri et al., 2011] and ridge regression via covariance perturbation [Smith et al., 2017].
4 For logistic regression, we leveraged the KDD-99 dataset [KDD, 1999] with d = 38 features, predicting whether network events can be classiﬁed as “normal" or “malicious".
For ridge regression, we used the Twitter dataset [Kawala et al., 2013] with d =features to predict log-popularity of posts.
In each case, we ran our experiments on n = 10 , ( y i , β T x i ) + λ || β
|| 22. [Figure] randomly sub-sampled data points. In order to guarantee bounded sensitivity, we normalized each data point to have unit (cid:96) 2 norm.
We note that this aspect differs from the experimentation conducted by Ligett et al.
[2017], who normalized by the maximum (cid:96) 2 norm, a non-private operation.
Experiments: For each task, we conducted two experiments.
We discuss the speciﬁc parameter settings for these experiments in Appendix F.
In the ﬁrst experiment, we plotted guaranteed (in the case of LNR ) or high-probability (in the case of BM ) privacy loss on the x-axis against average loss (either logistic or ridge) on the y-axis.
We conduct such a comparison as probability 1 privacy loss bounds cannot be provided for the Gaussian mechanism.
Likewise, adding a probability δ of minimally improves privacy loss for the Laplace mechanism.
We computed the average loss curve for each mechanism over 1,000 trials, and have included point-wise valid 95% conﬁdence intervals.
In the second experiment, we plotted the empirical privacy loss distributions for BM and LNR under the stopping conditions of loss being at most 0.41 for logistic regression and 0.025 for ridge regression.
For each mechanism, we evaluated this empirical distribution using three approaches for testing empirical loss: treating the training data as a held-out dataset, using AboveThreshold , and using our mechanism, ReducedAboveThreshold .
In AboveThreshold , we set the privacy parameter to be ﬁxed at (cid:15) = 0 .
In ReducedAboveThreshold , we took the sequence of privacy parameters to be the same as the sequence of privacy parameters used by BM and LNR .
We once again computed these empirical distributions over 1,000 runs of each mechanism.
Findings:
The ﬁndings of the two experiments are summarized in Figure 2 and Figure 3.
For both tasks, BM obtains signiﬁcant improvements in loss over LNR near the privacy loss level that was optimized for.
For both tasks, the privacy loss distribution for BM has lower median privacy loss than that of LNR .
In addition, the privacy loss distribution for BM is more tightly concentrated around the median, indicating more consistent performance.
The privacy loss distribution for LNR has a heavy tail, demonstrating that many runs do not attain the target loss until high privacy loss costs are incurred.
Comparing ReducedAboveThreshold and AboveThreshold , we see that the privacy loss distribution for ReducedAboveThreshold has higher variance than that of AboveThreshold .
However, ReducedAboveThreshold attains a signiﬁcantly lower median level of privacy loss when coupled with BM .
This latter point reﬂects the observations of Ligett et al.
[2017], who note that when AboveThreshold is used to determine stopping conditions on private data, it contributes the bulk of the privacy loss to the empirical distributions.
On the other hand, our ﬁgures demonstrate that
ReducedAboveThreshold results in a more mild privacy loss at target stopping conditions.
Conclusion
In this paper, we constructed the Brownian mechanism ( BM ), a novel approach to noise reduction that adds noise to a hidden parameter via a Brownian motion.
We not only precisely characterized the privacy loss of the Brownian mechanism, but also bounded it through applying machinery from continuous time martingale theory.
We then demonstrated how the utility of the iterates produced by
BM can be assessed on private data via ReducedAboveThreshold , a generalization of the classical
AboveThreshold algorithm.
This was itself accomplished by a continuous-time generalization of the original Laplace noise reduction ( LNR ) mechanism.
Last, we empirically demonstrated that BM outperforms LNR on common statistical tasks, such as regularized logistic and ridge regression.
We comment on several limitations and open problems related to our work.
We considered noise reduction mechanisms in the setting of one-shot privacy, in which only a single mechanism is run on private data.
Traditional composition results, such as those for ﬁxed privacy parameters [Dwork et al., 2010, Kairouz et al., 2015, Murtagh and Vadhan, 2016] or adaptively selected parameters [Rogers et al., 2016, Feldman and Zrnic, 2021, Whitehouse et al., 2022] are not directly applicable to algorithms satisfying ex-post privacy; additional machinery needs to be developed to handle composition in this case.
A naive approach to composition is possible, which involves summing the ex-post privacy guarantees of composed algorithms and summing the corresponding δ ’s, but we expect this approach to be loose.
Finally, noise reduction is currently only applicable to output perturbation methods; it remains open to see how to combine noise reduction with other prominent methods for private computation, such as objective perturbation.
Acknowledgements
AR acknowledges support from NSF DMS 1916320 and an ARL IoBT CRA grant.
Research reported in this paper was sponsored in part by the DEVCOM Army Research Laboratory under
Cooperative Agreement W911NF-17-2-0196 (ARL IoBT CRA).
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
ZSW and JW were supported in part by the NSF CNS2120667, a CyLab 2021 grant, a Google Faculty
Research Award, and a Mozilla Research Grant.
JW acknowledges support from NSF GRFP grants DGE1745016 and DGE2140739.
A Background on Martingale Concentration
In this section, we provide a background on the basics of martingale concentration needed throughout this paper.
Central to all results in this section is Ville’s inequality [Ville, 1939], which can be viewed as a time-uniform version of Markov’s inequality for martingales.
Lemma A.1 ( Ville’s Inequality [Ville, 1939]) .
Let ( X t ) t ≥ 0 be a nonnegative supermartingale with respect to some ﬁltration ( F t ) t ≥ 0 .
Then, for any conﬁdence parameter δ ∈ (0 , 1) , we have
[Equation]
While standard Brownian motion ( B t ) t ≥ 0 is not a nonnegative supermartingale, geometric Brownian motion given by Y λt := exp (cid:16) λB t − λ 2 2 t (cid:17) is a nonnegative martingale for any λ ∈ R , and hence
Lemma A.1 can be applied.
In fact, the probability in the lemma above becomes exactly δ when it is applied to a nonnegative martingale with continuous paths like Y λt .
From Ville’s inequality, the following line-crossing inequality for Brownian motion can be obtained.
Lemma A.2 (Line-Crossing Inequality) .
For δ ∈ (0 , 1) and a, b > 0 satisfying e − 2 ab =
δ , we have
P [Equation] at + [Equation]
= [Equation] A proof of the above fact can be found in any standard book on continuous time martingale theory [Le Gall, 2016, Durrett, 2019].
The above also follows from a special case of the more general time-uniform Chernoff bound presented in Howard et al.
The above inequality can be seen as optimizing the tightness of the time-uniform boundary at one pre-selected point in time.
However, due to the adaptive nature of the Brownian mechanism presented in Section 3, it is sometimes desirable to construct a time-uniform boundary which sacriﬁces tightness at a ﬁxed point in time to obtain greater tightness over all of time.
The method of mixtures provides one such approach for constructing tighter time-uniform boundaries [Kaufmann and Koolen, 2021, Howard et al., 2021].
We discuss this concept brieﬂy in the context of Brownian motion.
Observe that, since ( Y λt ) t ≥ 0 is a nonnegative martingale, for any probability measure π on R , the process ( X πt ) t ≥ 0 given by
[Equation]
is also nonnegative martingale.
By appropriately choosing the probability measure π and applying
Ville’s inequality, one obtains the following concentration inequality [Howard et al., 2021].
Lemma A.3 (Mixture Inequality) .
Let ρ > 0 and δ ∈ (0 , 1) be arbitrary.
Then,
[Equation]
We leverage Lemmas A.2 and A.3 to construct the privacy boundaries in Theorem 3.6 in Appendix B.
B Proofs From Section
Here, we prove the results from Section 3.
We start by showing that BM is in fact a noise-reduction mechanism, per the condition in Deﬁnition 2.3.
Proof of Lemma 3.2 .
For any initial value µ ∈ R d and any n ≥ 1 , let p µ 1: n denote the joint density of
B T n , . .
, B T 1 where ( B t ) t ≥ 0 is a Brownian motion started at µ and T n := T n ( x ) is a time function.
We have the decomposition
[Equation]
· T m −
T m (cid:33) .
[Figure]
This decomposition follows as B T 1 ∼ N ( µ, T 1 ) , and, given B T 1 , . . .
, B T m − 1 , we have that B T m ∼
N (cid:16) µ + T m
[Equation]
B
[Equation]
− [Equation] [Equation] m −(cid:17) , as B T m is conditionally distributed as a Brownian bridge.
Now, a straightforward calculation yields the equivalence
[Equation]
Hence, for any two mean vectors µ, µ (cid:48) ∈ R d , we can decompose the ratio of densities as
[Equation]
which is just the ratio between the density of a N ( µ, T n ) random variable evaluated at B T n and the density of a N ( µ (cid:48) , T n ) random variable evaluated at B T n , proving precisely the desired result.
We now prove Theorem 3.4, which gives a closed form characterization of the Brownian mechanism.
In what follows, we leverage the same notation for the density of Brownian motion as in the above proof.
Proof of Theorem 3.4 .
The second statement of the theorem is trivial and follows from our assumption of bounded (cid:96) 2 sensitivity.
Hence, we only prove the ﬁrst statement below.
Without loss of generality, and for the sake of simplicity, we can consider the function g ( y ) :
= f ( y ) − f ( x ) , as then g ( x ) = 0 .
Observe that, for y ∈ X , the random vector g ( y )
+ B t has Lebesgue density
[Equation]
Consequently, the privacy loss can be written as
[Equation]
Note that the last inequality follows from the fact that if ( B t ) t ≥ 0 is a d-dimensional Brownian motion and z ∈ R d is a unit vector under the (cid:96) 2 norm, then the process W t := (cid:104) z, B t (cid:105) is a standard Brownian motion.
Noting that g ( x (cid:48) )
=
f ( x (cid:48) ) − f ( x ) and that ( − W t ) t ≥ 0 is also a Brownian motion furnishes the result.
We now use the characterization of privacy loss in Theorem 3.4 alongside the time-uniform concentration results for continuous time martingales found in Appendix A to construct two general families of privacy boundaries.
We now prove Theorem 3.6.
Proof of Theorem 3.6 .
Recall from Theorem 3.4 that we have the following bound
[Equation]
where, for a random variable X , the random variable X + is given by max( X, 0) .
First, by leveraging
Lemma A.3, we see that, with probability at least 1 − δ , for all n ∈ N , we have
[Equation]
proving that ψ Mρ is a valid δ -privacy boundary.
Likewise, by Lemma A.2, we have that
[Equation]
showing ψ La,b is a valid δ -privacy boundary.
C Proofs From Section
In this appendix, we provide proofs of the results in Section 5.
We start by proving the privacy guarantees for ReducedAboveThreshold .
Proof of Theorem 5.1 .
For ReducedAboveThreshold as described in Algorithm 1, on the event { N ( x ) = n } , all information leaked about the underlying private dataset is contained in Alg 1: n ( x ) and α 1: n ( x ) , where α n ( x ) is deﬁned to be the n th bit output by ReducedAboveThreshold .
For any y ∈ X , let q y 1: n denote the joint density of (Alg 1: n ( y ) , α 1: n ( y )) , p y 1: n the marginal density of
Alg 1: n ( y ) , and p y 1: n ( · | · ) the conditional pmf of α 1: n ( y ) given the observed values of Alg 1: n ( y ) .
As such, for any neighboring datasets x ∼ x (cid:48) , on the event { N ( x ) = n } , the privacy loss of
ReducedAboveThreshold , denoted by L RAT ( x, x (cid:48) ) , is given by
[Equation]
where 0 n − 1 1 denotes the string of n − 1 0’s followed by a single 1.
In the last line we leverage the deﬁnition of the privacy loss between Alg 1: n ( x ) and Alg 1: n ( x (cid:48) ) and deﬁne
[Equation]
Now, to ﬁnish the result, it sufﬁces to prove that, for any n , L n ( x, x (cid:48) )
≤ E n (Alg 1: n − 1 ( x )) .
Without loss of generality, we can assume all thresholds take the same value τ across rounds, as we can always deﬁne the shifted function u (cid:48) n (Alg 1: n ( x ) , x ) :
= u n (Alg 1: n ( x ) , x ) − τ n + τ .
To prove our desired inequality, we proceed largely in the same way as the proof of AboveThreshold found in Lyu et al.
[2017], noting that conditioning on Alg 1: n ( x ) serves to ﬁx the utility functions u 1 (Alg 1 ( x ) , · ) , . . .
, u n (Alg 1: n ( x ) , · ) and the privacy levels E 1 , E 2 (Alg 1 ( x )) , . .
, E n (Alg 1: n − 1 ( x )) .
For simplicity, going forward, we refer to the former quantities as u 1 ( · ) , . . .
, u n ( · ) and the latter quantities just as (cid:15) 1 , . .
, (cid:15) n .
The only remaining caveat that we must take care in handling variable amount of noise on the threshold introduced by LNR .
Going forward, let P 1: n denote the conditional probability P ( · | Alg 1: n ( x )) .
First, observe that we can write the numerator of L n ( x, x (cid:48) ) [-asp x (cid:0) 
 n −-]as p
[Equation]
where g τ 1: n represents the density for the joint distribution of ( τ + Z (2∆ /(cid:15) m ))
nm =1 , where ( Z ( t ))
t ≥ η is as deﬁned in Equation (4).
We now need three inequalities.
The ﬁrst two are standard from the analysis of Lyu et al.
[2017], so we do not provide a proof.
The third inequality is a product of our novel ReducedAboveThreshold mechanism, and hence we provide a proof.
The inequalities are as follows:
For i < n and ﬁxed s i , P 1: n ( u i ( x ) + ξ i < s i ) ≤ P 1: n ( u i ( x (cid:48) )
+ ξ i < s i + ∆) , 2.
for i = n and any s n , P 1: n ( u n ( x ) + ξ n ≥ s n ) ≤ e (cid:15) n / 2 P 1: n ( u n ( x (cid:48) )
+ ξ n ≥ s n + ∆) , and 3.
for any s 1: n ∈ R n , g τ 1: n ( s 1 , . . .
, s n ) ≤ e (cid:15) n / 2 g τ 1:
n ( s 1 + ∆ , . . .
, s n + ∆) .
We now prove the third inequality.
We have that
[Equation]
where the ﬁrst equality follows from applying Bayes rule to the joint densities of the noisy thresholds, and the second equality follows from the fact that ( Z ( t )) forms a Markov process.
This in particular implies that the density conditional density given the n th threshold satisﬁes g a 1: n − 1 ( s 1 , . . .
, s n − 1 | s n ) =
g b 1: n − 1 ( s 1 , . . .
, s n − 1 | s n ) for all a, b ∈ R .
The last inequality follows from examining the ratio of densities of Lap( τ, 2∆ /(cid:15) n ) and Lap( τ − ∆ , 2∆ /(cid:15) n ) random variables.
Now, observe that by a simple shift of parameters we have
[Equation]
Plugging this in, we have
[Equation]
Rearranging furnishes the desired result.
We can also prove a corresponding utility guarantee for ReducedAboveThreshold .
As mentioned earlier, this utility guarantee is naive in the sense that it is derived from a union bound.
Thus, instead of plotting the utility guarantee in our experiments in Section 6, we instead plot empirically observed loss/accuracy.
Additionally, for the utility guarantee to hold, the sequence of privacy functions ( E n ) n ≥ 1 must be constant functions, i.e. E n = (cid:15) n for each n .
We now state the formal, high-probability utility guarantee in the following proposition.
Proposition C.1.
Let ( p n ) n ≥ 1 be a sequence of non-negative numbers
such that (cid:80) ∞
i =1 p i = 1 , and let γ ∈ (0 , 1) be a conﬁdence parameter.
Deﬁne the sequence of parameters ( η n ) n ≥ 1 by
[Equation]
Then, if N ( x ) is the time deﬁne in Theorem 5.1, with probability at least 1 − γ , we have u N ( x ) ( x ) ≥ τ N ( x ) − η N ( x ) .
Proof.
The above utility guarantee follows from applying two simple union bounds.
First, we have
P   (cid:91) n ≥
[Equation]
p
[Equation]
= 1 . Second, we have that [Equation] n η n 2∆ (cid:19) ≤ γ (cid:88) n ≥p n = 1 .
Thus, with probability at least 1 − γ , we have simultaneously for all n ≥ 1 that | ξ n | ≤ η n / 2 and | ζ n | ≤ η n / 2 .
Thus, with the same probability, on round N ( x ) , we have u N ( x ) ( x ) ≥ τ N ( x ) − η N ( x ) .
D Proofs From Section
We ﬁrst prove that the process deﬁned in Equation (4) has Laplace marginal distributions.
In what follows, we sometimes use the notation Z ( t ) interchangeably with Z t for convenience.
Theorem D.1.
Let ( Z t ) t ≥ η be the process deﬁend in Equation (4) .
Then, for any t ≥ η , we have
[Equation]
Proof.
Recall that if X ∼ Lap( s ) , then X has characteristic function ϕ s given by
[Equation]
Let φ denote the characteristic function of Z t − Z η .
Since Z η and Z t − Z η are independent, to show
Z t ∼ Lap( t ) , it sufﬁces to show that
[Equation]
Now, observe that the inhomogenous Poisson process ( P t ) t ≥ η can be written as ( (cid:101) P ( e t/ 2 ))
t ≥ log( η 2 ) where (cid:101) P is a homogeneous Poisson process with rate λ = 1 on [log( η 2 ) , ∞ ) .
In terms of the process (cid:101) P , we can consider the process ( (cid:101) Z t ) [Equation] 2 ) given by
[Equation]
, where (cid:101) T n := inf { t ≥ log( η 2 ) : (cid:101) P t ≥ n } and (cid:101) T 0 = log( η 2 ) .
It is easy to see that (cid:101)
Z (log( t 2 )) − (cid:101) Z (log( η 2 )) =
d Z t − Z η .
Leveraging this identity, it follows that we have φ
[Equation]
= ∞ (cid:88) n = ηt
[Equation]
In the above, f ( n ) ( u 1 , . . .
, u n ) :
= n !
[log( t 2 /η 2 )] n is the distribution of the order statistics ( U (1) , . .
, U ( n ) ) of n i.i.d.
random variables that are uniform on [log( η 2 ) , log( t 2 )] .
Essentially, what we have done is ﬁrst conditioned of the number of Poisson arrivals that occur in the interval [log( η 2 ) , log( t 2 )] .
Then, on the event { N ( t ) = n } , we condition again on the location of the n arrivals, which we know to be uniformly distributed across the time interval.
Once the arrival locations are known, we can compute the conditional characteristic function, which is the the product of characteristic functions as illustrated in the integral above.
Now, we show inductively that
[Equation]
The base case of n = 1 is trivially true.
Now, we have that
[Equation]
Leveraging this identity and picking up from the expression for φ ( λ ) in Equation (5), we have that
[Equation]
This proves the desired result.
The above proof can also be leveraged to show
that, for any ﬁnite ﬁxed sequence of times ( t n ) n ∈
[ K ] , ( Z ( t 1 ) , . . .
, Z ( t K )) has the same distribution as ( ζ 1 , . . .
, ζ K ) , where ( ζ n ) n ∈ [ K ] is the Laplace process associated with times ( t n ) n ∈
[ K ] as outlined in Equation (3).
This justiﬁes that the process ( Z ( t ))
t ≥ η is in fact a continuous time generalization of the aforementioned discrete time process.
E Skellam Noise Reduction
Lastly, we brieﬂy discuss how to generate a noise reduction mechanism for Skellam noise [Agarwal et al., 2021].
Recall that a random variable X has a Skellam distribution with parameters λ 1 and λif X
= d Y 1 − Y 2 , where Y 1 ∼ Poisson( λ 1 ) and Y 2 ∼ Poisson( λ 2 ) are independent Laplace random variables.
For succinctness, we write X ∼ Skell ( λ 1 , λ 2 ) .
Let ( P 1 ( t ))
t ≥ 0 and ( P 2 ( t ))
t ≥ 0 be two independent, homogeneous Poisson process with rates λ 1 and λ 2 respectively.
Observe that the continuous time process ( X t ) t ≥ 0 given by X t := P 1 ( t ) − P 2 ( t ) is clearly Markovian, has independent increments, and has X t ∼ Skell( tλ 1 , tλ 2 ) .
Thus, ( X t ) t ≥ 0 can be used to deﬁne a Skellam noise reduction mechanism by releasing ( f ( x ) + X T n ( x ) ) n ≥ 1 for some sequence of time functions ( T n ) n ≥ 1 .
F Additional Experimental Details
Parameter settings: We set the regularization parameter to be λ = 0 . 05 and note that the (cid:96) 2 and (cid:96) 1 -sensitivity for the output perturbation of logistic regression are respectively 2 nλ and 2 √ d nλ .
Likewise, for covariance perturbation in ridge regression, the (cid:96) 2 -sensitivities for privately releasing X T X and
X T y are both 2 . 0 , and the corresponding (cid:96) 1 -sensitivities for releasing these quantities are 2 .
d and 2 . [-
-]0 √ d respectively [Ligett et al., 2017, Chaudhuri et al., 2011].
We set the failure probability for BM to be δ = 10 − 6 , and in each task map privacy parameters ( (cid:15) n ) to times ( t n ) using the linear privacy boundary ψ La,b optimized for tightness at (cid:15) = 0 . 3 .
Optimizing privacy boundaries: We provide a high level description of how one may set the parameters associated with the privacy boundaries discussed in Theorem 3.6.
Let us consider the case of the mixture boundary ψ Mρ for illustrative purposes.
Suppose a data analyst desires that the ﬁnal level of privacy loss obtained by interacting with the
Brownian mechanism should be approximately (cid:15) .
Then, intuitively, the analyst should want to add the variance of the Gaussian noise added to be as small as possible when the privacy boundary takes value (cid:15) .
In mathematical notation, the analyst wants to ﬁnd a parameter ρ ∗ satisfying
[Equation]
where we note that the inverse function ( ψ Mρ ) − 1 exists as ψ Mρ is strictly increasing.
While this inverse has no closed form in general, the parameter ρ ∗ can be efﬁciently computed using several lines of code.
A similar, even more straightforward computation can be conducted for the linear privacy boundary.
Simulating Noise Reduction Mechanisms
We brieﬂy describe how a data analyst can produce samples from the Brownian mechanism and the Laplace noise reduction mechanism.
First, since T 1 ( x ) is a constant, we have BM 1 ( x ) ∼
N (
[Equation]
( x ) , T 1 ( x )) .
Then, given BM 1: m − 1 ( x ) , we have BM m ( x ) ∼ N
[Equation]
(cid:17) .
Since simulating the Brownian mechnaism only requires normal samples, it can be efﬁciently computed.
Second, to sample from LNR , one can ﬁrst generate the the points of arrival of the inhomogeneous
Poisson process ( P t ) t ≥ η up to time T 1 ( x ) .
Let T 1 , . .
, T N denote these arrival times, where we note that N , the number of arrivals up to time T 1 ( x ) , is a random variable.
Then, one can generate
Y m ∼ Lap( T m ) for m ≤ N .
From this information, the process ( Z t ) η ≤ t ≤ T 1 ( x ) can be readily computed, as in Equation (4).