Brownian Noise Reduction: Maximizing Privacy Subject to Accuracy Constraints
Anonymous Author(s)
AfﬁliationAddress email
Abstract
There is a disconnect between how researchers and practitioners handle privacyutility tradeoffs.
Researchers primarily operate from a privacy ﬁrst perspective,setting strict privacy requirements and minimizing risk subject to these constraints.
Practitioners often desire an accuracy ﬁrst perspective, possibly satisﬁed with thegreatest privacy they can get subject to obtaining sufﬁciently small error.
Ligettet al.
[2017] have introduced a “noise reduction” algorithm to address the latterperspective.
The authors show that by adding correlated Laplace noise and progres- sively reducing it on demand, it is possible to produce a sequence of increasinglyaccurate estimates of a private parameter while only paying a privacy cost for theleast noisy iterate released.
In this work, we generalize noise reduction to the setting of Gaussian noise, introducing the Brownian mechanism .
The Brownian mechanism works by ﬁrst adding Gaussian noise of high variance corresponding to the ﬁnal point of a simulated Brownian motion.
Then, at the practitioner’s discretion, noise is gradually decreased by tracing back along the Brownian path to an earlier time.
Our mechanism is more naturally applicable to the common setting of bounded (cid:96) 2 -sensitivity, empirically outperforms existing work on commonstatistical tasks, and provides customizable control of privacy loss over the entireinteraction with the practitioner.
We complement our Brownian mechanism with
ReducedAboveThreshold , a generalization of the classical AboveThreshold algorithm that provides adaptive privacy guarantees.
Overall, our results demonstratethat one can meet utility constraints while still maintaining strong levels of privacy.
Introduction
Over the past decade, differential privacy has seen industry-wide adoption as a means of protectingsensitive information [Erlingsson et al., 2014, Greenberg, 2016].
By injecting appropriate amountsof noise, differentially private algorithms allow the computation of population-level quantities ofinterest while guaranteeing individual-level privacy.
Of the private mechanisms used in industry,those relating to private empirical risk minimization (ERM) are perhaps the most impactful, in partdue to their application in machine learning tasks [Abadi et al., 2016, Song et al., 2013].
Researchershave developed many private ERM mechanisms, ranging from least squares minimzation [Sheffet,2017, Chaudhuri et al., 2011] to subsampled gradient descent [Abadi et al., 2016, Balle and Wang,2018, Wang et al., 2019].
Despite this vast literature, most existing results take the same broadapproach: they aim to minimize error (statistical risk) subject to strict privacy guarantees.
While thisstrict adherence to privacy constraints may be necessary in some applications, it often provides weakutility guarantees [Fienberg et al., 2010] and can make some learning tasks impossible
[Dwork et al.
Industry applications of differential privacy may desire an accuracy ﬁrst perspective, settingdesired risk requirements for models used in production.
Privacy may still be a desirable aspect ofcomputation, but it is by no means the only goal; minimizing risk may take center stage.
The main existing approach to this accuracy-oriented perspective on privacy was given by Ligettet al.
These authors introduce a noise reduction mechanism for gradually releasing a private,high-dimensional parameter.
By leveraging a Laplace-based Markov process [Koufogiannis et al.
,2017], they construct a mechanism for which the privacy loss of releasing arbitrarily many estimatesof a parameter only depends on the privacy loss of the least noisy parameter viewed.
This is incontrast to results about the composition of private algorithms, in which privacy degrades accordingto the total number of parameters witnessed [Dwork et al., 2010, Kairouz et al., 2015, Murtaghand Vadhan, 2016].
The authors also demonstrate how to privately query the utility of observedparameters on private data by coupling their Laplace-based mechanism with AboveThreshold , aclassical differentially private algorithm
[Dwork and Roth, 2014, Lyu et al., 2017].
While the above mechanism provides signiﬁcant privacy loss savings over a baseline method thatdoubles the privacy loss each round, Laplace noise is unﬁt for many settings in which (cid:96) 2 -sensitivity isused for calibrating noise.
Since converting from (cid:96) 2 -sensitivity to (cid:96) 1 -sensitivity 1 incurs a dimensiondependent cost, it is important to develop a noise reduction technique with Gaussian noise.
Contributions and paper outline.
We introduce the Brownian mechanism , a novel approach forprivately releasing a parameter vector subject to accuracy constraints.
The Brownian mechanism addscorrelated Gaussian noise to a risk-minimizing parameter through a Brownian motion.
Noise is theniteratively stripped by moving adaptively backwards along the random walk until a suitable stoppingcondition is met, such as meeting a target accuracy on a public dataset.
In Section 3, we deﬁne the
Brownian mechanism and characterize its privacy loss.
Using machinery from martingale theory, weconstruct privacy boundaries for the Brownian mechanism — upper bounds on privacy loss that holdsimultaneously with high probability.
In particular, the failure probability of these bounds does notdepend on the number of outcomes observed, overcoming a seeming need for a union bound facedby Ligett et al.
These privacy boundaries yield provable, high-probability bounds on privacyloss under data-dependent stopping conditions.
If private data is used to evaluate risk, then the data-dependent stopping conditions can themselves leakinformation.
To counter this, we introduce ReducedAboveThreshold in Section 5, a generalizationof the classical AboveThreshold algorithm for privately querying accuracy on sensitive data.
Weshow how to couple ReducedAboveThreshold and the Brownian mechanism so that a data analystonly ever incurs twice the privacy loss they would incur if they had queried accuracy on a publicdataset.
This is in contrast to the results in Ligett et al.
[2017], which note that the privacy loss of
AboveThreshold often dominates the privacy loss incurred from using noise reduction.
We empirically evaluate the Brownian mechanism and ReducedAboveThreshold in Section 6,ﬁnding that the Brownian mechanism can offer privacy loss savings over the Laplace noise reductionmethod introduced by Ligett et al.
In our view, these results demonstrate that the Brownianmechanism is a practical, intuitive mechanism for meeting accuracy requirements in private ERM.
Lastly, we derive other new mechanisms for noise reduction, of independent interest.
We generalizethe Laplace process of Koufogiannis et al.
[2017] to continuous time in Section 4, thus makingthe Laplace noise reduction mechanism of Ligett et al.
[2017] more ﬂexible and adaptive to datadependent privacy levels.
We also brieﬂy mention a noise reduction mechanism for Skellam noise in
Appendix E, a discrete distribution used in count queries [Agarwal et al., 2021].
Preliminaries
Differential privacy, privacy loss, and ex-post privacy.
An algorithm A : X → Y is ( (cid:15), δ ) -differentially private if, for any measurable set E ⊂ Y and any neighboring inputs x ∼ x (cid:48) ,
[Equation]
In the above [Dwork et al., 2006], ∼ denotes some arbitrary neighboring relation.
Typically x ∼ x (cid:48)indicates x and x (cid:48) differ in one entry, but any other relation sufﬁces.
While differential privacy hasproven itself a mainstay of private computation, condition (1) is too rigid to allow data analysts toachieve a minimum desired accuracy.
In other words, it embraces a privacy ﬁrst perspective, ﬁxing astrict condition in terms of parameters (cid:15) and δ that must be met.
We are interested in the accuracyﬁrst perspective, setting a target accuracy and correspondingly optimizing privacy parameters.
The above deﬁnition of differential privacy is qualitatively focused on bounding the informationtheoretic quantity of privacy loss [Dwork et al., 2006, 2010, Dwork and Roth, 2014].
Deﬁnition 2.1 ( Privacy Loss ) .
Let A : X → Y be an algorithm, and ﬁx neighbors x ∼ x (cid:48) in X .
Let p x and p x (cid:48) be the respective densities of A ( x ) and A ( x (cid:48) ) on the space Y with respect to somereference measure 2 .
Then, the privacy loss between A ( x ) and A ( x (cid:48) ) is the random variable
[Equation]
We think of A ( x ) as the true outcome, and L ( x, x (cid:48) ) measures how much more likely this outcomeis under the true input x versus an alternative x (cid:48) .
Privacy loss provides a probabilistic deﬁnition ofprivacy.
Namely, A is ( (cid:15), δ ) -probabilistically differentially private if, for all neighbors x ∼ x (cid:48) ,
[Equation]
While probabilistic differential privacy is not equivalent to differential privacy [Kasiviswanathanand Smith, 2014], ( (cid:15), δ ) -probabilistically differential privacy implies ( (cid:15), δ ) -differential privacy.
Probabilistic differential privacy emerged as a means for studying privacy composition, and has beenleveraged in proving many results [Kairouz et al., 2015, Murtagh and Vadhan, 2016, Rogers et al.
,2016, Whitehouse et al., 2022].
A natural extension of privacy to the accuracy-oriented regime isex-post privacy, which allows the bound in condition (2) to depend the observed algorithm output.
Deﬁnition 2.2 (Ligett et al.
Let A : X → Y be an algorithm and E : Y → R ≥ 0 a function.
We say A is ( E , δ ) -ex-post private if, for any neighboring inputs x ∼ x (cid:48) , we have
[Equation]
While any algorithm is trivially ex-post private with E ( A ( x )) := ∞ , the goal is to make E ( A ( x ))
assmall as possible.
We describe theoretical tools for obtaining ex-post privacy guarantees in Section 3,and empirically compute the ex-post privacy distributions of various mechanisms in Section 6.
Background on Noise Reduction.
Heuristically, a noise reduction mechanism allows a data analystto view multiple, increasingly accurate estimates of a risk minimizing parameter while only payingan ex-post privacy cost for the least noisy iterate observed.
More formally, let M : X → Y ∞ be anyalgorithm mapping databases to sequences of outputs.
The indices of the sequence denote the roundsof interaction, with smaller indices indicating greater noise.
Let M n : X → Y give the n th elementof the sequence and M 1: n : X → Y n the ﬁrst n elements.
Deﬁnition 2.3 ( Noise Reduction Mechanism ) .
We say M : X → Y ∞ is a noise reduction mechanism if, for any n ≥ 1 and any neighboring datasets x ∼ x (cid:48) , we have
[Equation]
where L 1: n ( x, x (cid:48) ) denotes the privacy loss between M 1: n ( x ) and M 1: n ( x (cid:48) ) and L n ( x, x (cid:48) ) is theprivacy loss between M n ( x ) and M n ( x (cid:48) ) .
The only noise reduction mechanism in the literature uses a Markov process with Laplacemarginals [Koufogiannis et al., 2017] to gradually release a sensitive parameter [Ligett et al., 2017].
As originally presented, this Laplace Noise Reduction mechanism is nonadaptive, requiring a dataanalyst to ﬁx a ﬁnite sequence of privacy parameters ( (cid:15) n ) n ∈
[ K ] in advance.
Instead of presenting thismethod as background, we describe it in Section 4, in which we construct an adaptive generalization ofthis mechanism.
We then leverage this generalization as a subroutine in ReducedAboveThreshold ,a generalization of AboveThreshold with adaptive privacy guarantees.
Background on Brownian Motion.
We now provide a brief background on Brownian motion,perhaps the best-known example of a continuous time stochastic process [Le Gall, 2016].
Deﬁnition 2.4.
A continuous time real-valued process ( B t ) t ≥ 0 is called a standard Brownian motionif (1) B 0 = 0 , (2) ( B t ) t ≥ 0 has continuous sample paths, (3) ( B t ) has independent increments, i.e.
B t + s − B s is independent of B s for all s, t ≥ 0 , and (4) B t ∼ N (0 , t ) for all t ≥ 0 .
We say a process ( B t ) t ≥ 0 is a d -dimensional standard Brownian motion if each coordinate process isan independent standard Brownian motion.
We use many properties of Brownian motion to construct the Brownian mechanism and analyze itsprivacy loss in Section 3.
One important property of Brownian motion is that it is a continuous timemartingale.
This property allow us to use time-uniform supermartingale concentration to characterizeand bound the privacy loss of the Brownian mechanism at data-dependent stopping times [Howardet al., 2020, 2021].
We do not go into detail about martingale concentration in this backgroundsection, but rather defer it to Appendix A. Additionally, ( B t ) t ≥ 0 is a Markov process.
This tells usthat if we inspect the Brownian motion at times 0 ≤ t 1 < t 2 < · · · < t n , then B t 2 , . . .
, B t n can beviewed as a randomized post-processing of B t 1 that does not depend on B s for any s < t 1 .
Thisproperty allows us to show that the privacy loss of the Brownian mechanism — which adds noise to aparameter via a Brownian motion — only depends on the least noisy parameter observed.
The Brownian Mechanism: a Gaussian Noise Reduction Mechanism
The Brownian mechanism works by simulating a Brownian motion starting at some multivariateparameter; this parameter should be thought of as the risk-minimizing output if there were no privacyconstraints.
The data analyst ﬁrst observes the random walk at some large time.
Then, if so desired,the analyst “rewinds” time to an earlier point on the Brownian path, reducing noise to obtain a moreaccurate estimate.
Due to the Markovian nature of Brownian motion, the analyst will only pay aprivacy cost proportional to variance of the random walk at the earliest inspected time.
In whatfollows, when we refer to a sequence ( T n ) n ≥ 1 of time functions , we mean a sequence of functions T n : ( R d ) n − 1 → R ≥ [-
 suchthat,-]0 such that, for all n and β 1: n ∈ ( R d ) n , T n +1 ( β 1: n ) ≤ T n ( β 1: n − 1 ) .
Deﬁnition 3.1.
Let f : X → R d be a function and ( T n ) n ≥ 1 a sequence of time functions.
Let( B t ) t ≥ 0 be a standard d -dimensional Brownian motion.
The Brownian mechanism associated with fand ( T n ) n ≥ 1 is the algorithm BM : X → ( R d ) ∞ given by
[Equation]
where we set T n ( x ) :
= T n (cid:0) f
( x ) + B T 1 ( x ) , . . . , f ( x ) + B T n − 1 ( x ) (cid:1) with T 1 ( x ) being constant.
We have chosen T n ( x ) as indexing notation to denote dependence on x , even if this is only throughobserved parameters.
In the context of ERM, one can think of f as computing a risk minimizingparameter associated with a private dataset x ∈ X .
The data analyst uses T n along with the ﬁrst n − 1iterates to determine how far to rewind time to obtain the n th iterate.
Due to the Markovian nature of Brownian motion, we get the following lemma.
We include a proof in Appendix B for completeness.
Lemma 3.2.
Let x ∼ x (cid:48) be neighbors.
For any n ≥ 1 , let L BM1: n ( x, x (cid:48) )
denote the privacy loss between
BM 1: n ( x ) and BM 1: n ( x (cid:48) ) and L BM n ( x, x (cid:48) )
the privacy loss between BM n ( x ) and BM n ( x (cid:48) ) .
Then,
L BM1: n ( x, x (cid:48) )
=
L BM n ( x, x (cid:48) ) .
Lemma 3.2 just tells us that the Brownian mechanism is a noise reduction mechanism, i.e. that theprivacy lost by viewing the ﬁrst n iterates is exactly the privacy lost by viewing the n th iterate inisolation.
Thus, we can identify L BM1: n ( x, x (cid:48) ) with L BM n ( x, x (cid:48) ) going forward.
The Brownian mechanism, as deﬁned above, produces an inﬁnite sequence of parameters.
In practice,a data analyst will only view ﬁnitely many iterates, stopping when some utility condition has beenmet or a minimum privacy level is reached.
We introduce stopping functions to model how a dataanalyst adaptively interacts with noise reduction mechanisms.
Deﬁnition 3.3 ( Stopping Function ) .
Let M : X → Y ∞ be a noise reduction mechanism.
Forx ∈ X , let ( F n ( x )) n ∈ N be the ﬁltration given by F n ( x ) :
= σ ( M i ( x ) : i ≤ n ) .
3 A function
N : Y ∞ → N is called a stopping function if for any x ∈ X , N ( x ) :
= N ( M ( x )) is a stopping timewith respect to ( F n ( x )) n ≥ 1 .
A stopping function N is a rule used to decide when to stop viewing parameters that only depends onthe observed iterates of the noise reduction mechanism.
N could heuristically be “stop at the ﬁrsttime a parameter achieves an accuracy of 95% on a held-out dataset.”
If a data analyst uses a stoppingfunction alongside BM , per Deﬁnition 2.3, the privacy loss accrued upon stopping is L BM N ( x ) ( x, x (cid:48) ) .
In some cases, an analyst may want the stopping function to depend on the underlying private datasetthrough more than just the released parameters, e.g. they may want their rule to be “stop at the ﬁrsttime a parameter achieves an accuracy of 95% on the private dataset.”
In this case, additional privacymay be lost due to observing N ( x ) .
We detail how to handle this more subtle case in Section 5.
Thefollowing theorem characterizes the privacy loss of the Brownian mechanism.
Theorem 3.4.
Let BM be the Brownian mechanism associated with ( T n ) n ≥ 1 and a function f .
Forneighbors x ∼ x (cid:48) , the privacy loss between BM 1: n ( x ) and BM 1: n ( x (cid:48) ) is given by
[Equation]
where ( W t ) t ≥ 0 is a standard, univariate Brownian motion.
Suppose f has (cid:96) 2 -sensitivity at most ∆ 2 .
Then, letting a + := max(0 , a ) , we have
[Equation]
Theorem 3.4 also holds when a deterministic time n is replaced by N ( x ) , where N is a stoppingfunction.
The above theorem can be viewed as a process-level equivalent of the well-known fact thatthe privacy loss of the Gaussian mechanism has an uncentered Gaussian distribution [Balle and Wang,2018].
We prove the Theorem 3.4 in Appendix B. Given the clean characterization of privacy lossabove, we now show how to construct high-probability, time-uniform privacy loss bounds.
We deﬁneprivacy boundaries , which map the variance of BM to high-probability bounds on privacy loss.
Deﬁnition 3.5.
A function ψ : R ≥
→ R ≥ 0 is a δ -privacy boundary for the Brownian mechanismassociated with time functions ( T n ) n ≥ 1 if for any neighboring datasets x ∼ x (cid:48) , we have
[Equation]
Since the privacy loss of BM is a deterministic function of a Brownian motion, we can apply resultsfrom martingale theory to construct general families of privacy boundaries.
Theorem 3.6.
Assume the same setup as in Theorem 3.4.
Let δ > 0 and f be a function with(cid:96) 2 -sensitivity ∆ 2 .
The following classes of functions form δ -privacy boundaries.
(Mixture boundary)
For any ρ > 0 , ψ Mρ given by
[Equation]
(Linear boundary)
For any a, b > 0 such that 2 ab = log(1 /δ ) , ψ La,b given by
[Equation]
We prove Theorem 3.6 in Appendix B. In the same appendix, we plot the boundaries in Figure 4.
Privacy boundaries serve a dual purpose for the Brownian mechanism.
First, since time-uniformconcentration bounds are valid at stopping times [Howard et al., 2021], privacy boundaries provideex-post privacy guarantees.
Second, in many settings, it may be more natural for a data analyst toadaptively specify target privacy levels instead of noise levels.
By inverting privacy boundaries, dataanalysts can compute the proper amount of noise to remove at each step to meet target privacy levels.
We make the above precise in Corollary 3.7.
In what follows, when we refer to a sequence ( E n ) n ≥ 1of privacy functions , we mean a sequence of functions E n : ( R d ) n − 1 → R ≥ 0 such that, for all n andβ 1: n ∈ ( R d ) n , E n +1 ( β 1: n ) ≥ E n ( β 1: n − 1 ) .
Corollary 3.7.
Let N be a stopping function, as in Deﬁnition 3.3.
If ψ is a δ -privacy boundary for
BM , we have
[Equation]
i.e. the algorithm BM 1: N ( · ) ( · ) is (cid:0) ψ ( T N ( · ) ( · )) , δ (cid:1)
-ex post private, where ( · ) denotes a positionalargument for an input x ∈ X .
Further, let ( E n ) n ≥ 1 be a sequence of privacy functions, and deﬁne
[Equation] ≥ 0 : ψ ( t ) ≥ E n ( β 1: n − 1 ) } .
Then BM 1: N ( · ) ( · ) is ( E N ( · ) ( · ) , δ ) -ex post private, where E n ( x ) is deﬁned analogously to T n ( x ) .
Again, N should be thought of as a stopping rule based on parameter accuracy.
E n should be thoughtof as a rule for choosing the n th privacy parameter given BM 1: n − 1 ( x ) .
An Adaptive, Continuous-Time Extension of Laplace Noise Reduction
Here, we generalize the original noise reduction mechanism of Ligett et al.
[2017], which will be usedas a subroutine in Algorithm 1 in the following section.
We ﬁrst describe the original Laplace-based Markov process of Koufogiannis et al.
Fix any positive integer K and any ﬁnite, increasingsequence of times ( t n ) n ∈
[ K ] .
Let ( ζ n )
Kn =0 be the d -dimensional process given by ζ 0 = 0 andζ n =
[Equation]
with probability (cid:16) t n − 1 t n (cid:17)ζ n − 1 + Lap( t n ) otherwise.
Koufogiannis et al.
[2017] show that ζ n ∼ Lap( t n ) and that ( ζ n )
Kn =0 is Markovian.
Ligett et al.[2017] use the above process to construct a noise reduction mechanism.
Namely, they deﬁne thethe Laplace Noise Reduction mechanism associated with f : X → R d and ( t n ) n ∈
[ K ] to be thealgorithm LNR : X → ( R d ) K given by LNR( x ) :
= ( f ( x ) + ζ K , . . .
, f ( x ) + ζ 1 ) .
If t n := ∆ 1 /(cid:15) n ,then releasing n th component LNR n ( x ) in isolation is equivalent to running the classical Laplacemechanism with privacy level (cid:15) n .
We now extend the process ( ζ n ) n ∈ [ K ] to a continuous time process with the same ﬁnite-dimensionaldistributions.
Let η > 0 be arbitrary, and let ( P t ) t ≥ η be an inhomogeneous Poisson process withintensity function λ ( t ) :
= 2 t .
For n ≥ 1 , let T n :=
inf { t ≥ η : P t ≥ n } be the n th jump of ( P t ) t ≥ ηand set T 0 :
=
η .
Noting that P t must be a nonnegative integer, deﬁne the process ( Z t ) t ≥ η by
[Equation]
It is immediate that ( Z t ) t ≥ η is Markovian.
We show in Appendix D that Z t ∼ Lap( t ) .
With ( Z t ) t ≥ η ,one can make LNR fully adaptive, meaning that the times ( t n ) n ∈ [ K ] at which it is invoked need notbe prespeciﬁed, and can depend on the underlying input database x by using time functions.
Deﬁnition 4.1.
Let f : X → R d be a function and ( T n ) n ≥ 1 a sequence of time functions.
Let( Z t ) t ≥ η be the process deﬁned in Equation (4) .
The Laplace noise reduction mechanism associatedwith f and ( T n ) n ≥ 1 is the algorithm LNR : X → ( R d ) ∞ given by
[Equation]
If the analyst would prefer instead to specify privacy functions ( E n ) n ≥ 1 , they can do so by leveragingthe corresponding time functions T n ( x ) :
=
∆ 1 / E n ( x ) , where E n ( x ) is deﬁned analogously to
T n ( x ) .
We leverage LNR in our experiments in Section 6 and the process ( Z t ) t ≥ 0 as a subroutine inconstructing ReducedAboveThreshold .
LNR admits the following trivial ex-post privacy guarantee.
Proposition 4.2.
Let LNR be associated with ( T n ) n ≥ 1 and a function f , and suppose f has (cid:96) 1sensitivity ∆ 1 .
If N is stopping function, the algorithm LNR 1: N ( · ) ( · ) is (∆ 1 /T N ( · ) ( · ) , 0) -ex postprivate.
Privately Checking if Accuracy is Above a Threshold
In Section 3 we presented the Brownian mechanism, characterized its privacy loss, and showed howto obtain ex-post privacy guarantees for arbitrary stopping functions.
In particular, these stoppingfunctions could be based on the accuracy of the observed iterates on public held-out data.
However, one may desire to privately check the accuracy of observed iterates on the dataset x ∈ X .
Ligett et al.
[2017] were able to accomplish this goal by coupling LNR with AboveThreshold , aclassical algorithm for privately answering threshold queries [Dwork and Roth, 2014].
In the contextof ERM, AboveThreshold iteratively checks if the empirical risk of each parameter is below atarget threshold, stopping at the ﬁrst such occurrence.
The downside to AboveThreshold is that itrequires a preﬁxed privacy level.
In empirical studies, Ligett et al. [2017] found this ﬁxed privacycost dominated the ex-post privacy guarantees, showing little beneﬁt to using noise reduction.
Below, we construct ReducedAboveThreshold , a generalization of AboveThreshold which provides ex-post privacy guarantees.
We show how to couple BM with ReducedAboveThreshold toobtain tighter ex-post privacy guarantees than coupling with AboveThreshold would permit.
Inparticular, if BM is run using parameters ( (cid:15) n ) n ≥ 1 and ReducedAboveThreshold indicates the N thparameter obtains sufﬁciently high accuracy, the privacy loss of the net procedure will be at most2 (cid:15) N — only twice the privacy loss that would be accrued by testing on public data.
Algorithm 1 ReducedAboveThreshold (via Laplace Noise Reduction)
Require:
Algorithm Alg :
X → Y ∞ , parameter (cid:15) max > 0 , threshold τ , database x ∈ X , utility u : Y × X → R where u ( β, · ) is ∆ -sensitive ∀ β , privacy functions ( E n ) n ≥ 1 with E n ≤ (cid:15) max ∀ n .
for n ≥ 1 do (cid:15) n := E n (Alg 1: n − 1 ( x )) , T n := 2∆ /(cid:15) n ζ n := Z T n , where ( Z t ) t ≥ η in Eq.
(4) deﬁnes the LNR mechanism with η := 2∆ /(cid:15) max .
ξ n ∼ Lap (cid:16) 4∆ (cid:15) n (cid:17) if u (Alg n ( x ) , x ) + ξ n ≥ τ + ζ n then
Print 1 and HALT else
Printτ should be seen as a target accuracy, Alg as a mechanism for releasing a parameter (e.g. BM , LNR ),and u as evaluating the accuracy of Alg n ( x ) on x . (cid:15) max is an arbitrarily large maximum privacy level,used to prevent the user from examining ( Z t ) at arbitrarily small times.
The above generalizes tosequences of thresholds ( τ n ) n ≥ 1 and sequences ( u n ) n ≥ 1 of functions u n : Y n × X → R that are∆ -sensitive in their second argument, but the added generality yields only marginal beneﬁts.
When
E n = (cid:15) for all n , Algorithm 1 recovers AboveThreshold as a special case.
The intuition behind
ReducedAboveThreshold is that by gradually removing Laplace noise from the threshold, a dataanalyst can ensure that privacy of the whole procedure only depends on the magnitude of Laplacenoise added when the algorithm halts.
The following characterizes the privacy loss of Algorithm 1.
Theorem 5.1.
For any n ≥ 1 and neighboring datasets x ∼ x (cid:48) , let L Alg1: n ( x, x (cid:48) ) denote the privacybetween Alg 1: n ( x ) and Alg 1: n ( x (cid:48) ) .
For any x ∈ X , deﬁne N ( x ) to be the ﬁrst round where
ReducedAboveThreshold run on input x ∈ X outputs 1, that is
N ( x ) := inf { n ≥ 1 : ReducedAboveThreshold n ( x ) = 1 } .
Then, the privacy loss between ReducedAboveThreshold( x ) and ReducedAboveThreshold( x (cid:48) ) ,denoted L RAT ( x, x (cid:48) ) , is bounded by
L RAT ( x, x (cid:48) ) ≤ L Alg1: N ( x ) ( x, x (cid:48) ) + E N ( x ) (Alg 1: N ( x ) − 1 ( x )) .
We prove Theorem 5.1 in Appendix C, where we also provide a utility guarantee for
ReducedAboveThreshold .
This utility guarantee, much like the utility guarantee for
AboveThreshold , is in practice weak as it derives from a union bound.
Using Theorem 5.1, we cansimply choose Alg = BM as a means of adaptively generating parameters.
The following corollary,which follows immediately from the above theorem, provides the ex-post privacy guarantees ofcombining ReducedAboveThreshold and BM .
Corollary 5.2.
Let BM be the Brownian mechanism associated with a function f , decreasing timefunctions ( T n ) n ≥ 1 , and a a δ -privacy boundary ψ .
Let ReducedAboveThreshold be run withprivacy functions ( ψ ( T n ))
n ≥ 1 , threshold τ , and algorithm BM .
Then, ReducedAboveThreshold is 280 (cid:0) 2 ψ ( T N ( · ) ( · )) , δ (cid:1)
-ex post private.
Experiments
[Figure] [Figure]
or (right) via ReducedAboveThreshold . Choice of tasks: We compare the performance of BM and LNR on the tasks of regularized logisticregression via output perturbation [Chaudhuri et al., 2011] and ridge regression via covarianceperturbation [Smith et al., 2017].
4 For logistic regression, we leveraged the KDD-99 dataset [KDD, The regularized β λ β 1999] with d = 38 features, predicting whether network events can be classiﬁed as “normal" or“malicious".
For ridge regression, we used the Twitter dataset [Kawala et al., 2013] with d = 77features to predict log-popularity of posts.
In each case, we ran our experiments on n = 10 , 000randomly sub-sampled data points.
In order to guarantee bounded sensitivity, we normalized eachdata point to have unit (cid:96) 2 norm.
We note that this aspect differs from the experimentation conductedby Ligett et al.
[2017], who normalized by the maximum (cid:96) 2 norm, a non-private operation.
[-Parameter settings: We set the regularization parameter to be λ = 0 . 05 and note that the (cid:96) 2 and(cid:96) 1 -sensitivity for the output perturbation of logistic regression are respectively 2 nλ and 2 √ d nλ . Likewise,for covariance perturbation in ridge regression, the (cid:96) 2 -sensitivities for privately releasing X T X and X T y are both 2 . 0 , and the corresponding (cid:96) 1 -sensitivities for releasing these quantities are 2 . 
 d and2 . 
-]Experiments: For each task, we conducted two experiments.
We discuss the speciﬁc parametersettings for these experiments in Appendix F.
In the ﬁrst experiment, we plotted guaranteed (in thecase of LNR ) or high-probability (in the case of BM ) privacy loss on the x-axis against averageloss (either logistic or ridge) on the y-axis.
We conduct such a comparison as probability 1 privacyloss bounds cannot be provided for the Gaussian mechanism.
Likewise, adding a probability δ ofminimally improves privacy loss for the Laplace mechanism.
We computed the average loss curvefor each mechanism over 1,000 trials, and have included point-wise valid 95% conﬁdence intervals.
In the second experiment, we plotted the empirical privacy loss distributions for BM and LNR underthe stopping conditions of loss being at most 0.41 for logistic regression and 0.025 for ridge regression.
For each mechanism, we evaluated this empirical distribution using three approaches for testingempirical loss: treating the training data as a held-out dataset, using AboveThreshold , and usingour mechanism, ReducedAboveThreshold .
In AboveThreshold , we set the privacy parameter tobe ﬁxed at (cid:15)
= 0 .
In ReducedAboveThreshold , we took the sequence of privacy parameters to bethe same as the sequence of privacy parameters used by BM and LNR .
We once again computedthese empirical distributions over 1,000 runs of each mechanism.
Findings:
The ﬁndings of the two experiments are summarized in Figure 2 and Figure 3.
For bothtasks, BM obtains signiﬁcant improvements in loss over LNR near the privacy loss level that wasoptimized for.
For both tasks, the privacy loss distribution for BM has lower median privacy loss thanthat of LNR .
In addition, the privacy loss distribution for BM is more tightly concentrated aroundthe median, indicating more consistent performance.
The privacy loss distribution for LNR has aheavy tail, demonstrating that many runs do not attain the target loss until high privacy loss costsare incurred.
Comparing ReducedAboveThreshold and AboveThreshold , we see that the privacyloss distribution for ReducedAboveThreshold has higher variance than that of AboveThreshold .
However, ReducedAboveThreshold attains a signiﬁcantly lower median level of privacy loss whencoupled with BM .
This latter point reﬂects the observations of Ligett et al.
[2017], who note thatwhen AboveThreshold is used to determine stopping conditions on private data, it contributes thebulk of the privacy loss to the empirical distributions.
On the other hand, our ﬁgures demonstrate that
ReducedAboveThreshold results in a more mild privacy loss at target stopping conditions.
Conclusion
In this paper, we constructed the Brownian mechanism ( BM ), a novel approach to noise reductionthat adds noise to a hidden parameter via a Brownian motion.
We not only precisely characterizedthe privacy loss of the Brownian mechanism, but also bounded it through applying machinery fromcontinuous time martingale theory.
We then demonstrated how the utility of the iterates produced by
BM can be assessed on private data via ReducedAboveThreshold , a generalization of the classical
AboveThreshold algorithm.
This was itself accomplished by a continuous-time generalization ofthe original Laplace noise reduction ( LNR ) mechanism.
Last, we empirically demonstrated that BMoutperforms LNR on common statistical tasks, such as regularized logistic and ridge regression.
We comment on several limitations and open problems related to our work.
We considered noisereduction mechanisms in the setting of one-shot privacy, in which only a single mechanism is run onprivate data.
Traditional composition results, such as those for ﬁxed privacy parameters [Dwork et al.
,2010, Kairouz et al., 2015, Murtagh and Vadhan, 2016] or adaptively selected parameters [Rogerset al., 2016, Feldman and Zrnic, 2021, Whitehouse et al., 2022] are not directly applicable to algorithms satisfying ex-post privacy; additional machinery needs to be developed to handle compositionin this case.
A naive approach to composition is possible, which involves summing the ex-postprivacy guarantees of composed algorithms and summing the corresponding δ ’s, but we expect thisapproach to be loose.
Finally, noise reduction is currently only applicable to output perturbationmethods; it remains open to see how to combine noise reduction with other prominent methods forprivate computation, such as objective perturbation.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] We construct the Brownian mechanism is Section 3,show how to query utility on private datasets in a manner that provides adaptiveprivacy guarantees in Section 5, and conduct experiments showing the Brownianmechanism outperforms Laplace noise reduction in Section 6 .
b
) Did you describe the limitations of your work?
[Yes] In the Conclusion (Section 7)we discuss how traditional composition results for differential privacy do not directly apply to mechanisms with ex-post privacy guarantees.
We leave it as futurework to address composition in this setting.
Additionally, in the same section, wemention noise reduction is only applicable to output perturbation methods.
Weleave it as future work to generalize to other modes of private computation, suchas objective perturbation.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
There
are no negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All theoretical
results have a full set of assumptions stated.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs of all
non-trivial results are fully contained in the appendices.
Appendix B contains
the proofs for Section 3, Appendix C contains the proofs for Section 5, and Ap-
pendix D contains proofs for Section 4.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code is
commented and contained in the supplement.(b)
Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] These details are included in the paper and code.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[Yes] We included 95% conﬁdence intervals on the
relevant experiments.
See the graphs in Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A] Experiments were lightweight
and run on a laptop.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The datasets are
properly attributed in the Bibliography.
(b) Did you mention the license of the assets?
[N/A]
The datasets leveraged in this
work are public, and hence do not require licensing.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No] we only use well-known, broadly used,
publicly available datasets, and there is no known offensive content or personal
information to the best of our knowledge.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Background on Martingale Concentration
In this section, we provide a background on the basics of martingale concentration needed throughoutthis paper.
Central to all results in this section is Ville’s inequality [Ville, 1939], which can be viewedas a time-uniform version of Markov’s inequality for martingales.
Lemma A.1 ( Ville’s Inequality [Ville, 1939]) .
Let ( X t ) t ≥ 0 be a nonnegative supermartingale withrespect to some ﬁltration ( F t ) t ≥ 0 .
Then, for any conﬁdence parameter δ ∈ (0 , 1) , we have
[Equation]
While standard Brownian motion ( B t ) t ≥ 0 is not a nonnegative supermartingale, geometric Brownianmotion given by Y λt := exp (cid:16) λB t − λ 2 2 t (cid:17) is a nonnegative martingale for any λ ∈ R , and hence
Lemma A.1 can be applied.
In fact, the probability in the lemma above becomes exactly δ when itis applied to a nonnegative martingale with continuous paths like Y λt .
From Ville’s inequality, thefollowing line-crossing inequality for Brownian motion can be obtained.
Lemma A.2 (Line-Crossing Inequality) .
For δ ∈ (0 , 1) and a, b > 0 satisfying e − 2 ab =
δ , we have
P ( ∃ t ≥ 0 : B t ≥ at + b ) = δ.
A proof of the above fact can be found in any standard book on continuous time martingale theory [Le Gall, 2016, Durrett, 2019].
The above also follows from a special case of the more generaltime-uniform Chernoff bound presented in Howard et al.
The above inequality can be seen as optimizing the tightness of the time-uniform boundary at onepre-selected point in time.
However, due to the adaptive nature of the Brownian mechanism presentedin Section 3, it is sometimes desirable to construct a time-uniform boundary which sacriﬁces tightnessat a ﬁxed point in time to obtain greater tightness over all of time.
The method of mixtures provides one such approach for constructing tighter time-uniform boundaries [Kaufmann and Koolen, 2021, Howard et al., 2021].
We discuss this concept brieﬂy in thecontext of Brownian motion.
Observe that, since ( Y λt ) t ≥ 0 is a nonnegative martingale, for anyprobability measure π on R , the process ( X πt ) t ≥ 0 given by
[Equation]
is also nonnegative martingale.
By appropriately choosing the probability measure π and applying
Ville’s inequality, one obtains the following concentration inequality [Howard et al., 2021].
Lemma A.3 (Mixture Inequality) .
Let ρ > 0 and δ ∈ (0 , 1) be arbitrary.
Then,
[Equation]
We leverage Lemmas A.2 and A.3 to construct the privacy boundaries in Theorem 3.6 in Appendix B.
B Proofs From Section
Here, we prove the results from Section 3.
We start by showing that BM is in fact a noise-reductionmechanism, per the condition in Deﬁnition 2.3.
Proof of Lemma 3.2 .
For any initial value µ ∈ R d and any n ≥ 1 , let p µ 1: n denote the joint density of
B T n , . .
, B T 1 where ( B t ) t ≥ 0 is a Brownian motion started at µ and T n := T n ( x ) is a time function. We have the decompositionp µ 1: n ( B T n , . . . , B T 1 )
[Equation]
exp
[Equation]
[Figure]
with δ 10 − 6 . The directly plots the bounds Theorem boundaries, variance necessary to meet a target privacy level.
This decomposition follows as B T 1 ∼ N ( µ, T 1 ) , and, given B T 1 , . . . , B T m − 1 , we have that B T m ∼ N (cid:16) µ + T m T
[Equation]
(cid:17) , as B T m is conditionally distributed as a Brownianbridge. Now, a straightforward calculation yields the equivalence
[Equation]
Hence, for any two mean vectors µ, µ (cid:48) ∈ R d , we can decompose the ratio of densities as
[Equation]
which is just the ratio between the density of a N ( µ, T n ) random variable evaluated at B T n and thedensity of a N ( µ (cid:48) , T n ) random variable evaluated at B T n , proving precisely the desired result.
We now prove Theorem 3.4, which gives a closed form characterization of the Brownian mechanism.
In what follows, we leverage the same notation for the density of Brownian motion as in the aboveproof.
Proof of Theorem 3.4 .
The second statement of the theorem is trivial and follows from our assumption of bounded (cid:96) 2 sensitivity.
Hence, we only prove the ﬁrst statement below.
Without loss of generality, and for the sake of simplicity, we can consider the function g ( y ) :
=f ( y ) − f ( x ) , as then g ( x ) = 0 .
Observe that, for y ∈ X , the random vector g ( y )
+ B t has Lebesguedensity
[Equation]
Consequently, the privacy loss can be written as
[Equation]
Note that the last inequality follows from the fact that if ( B t ) t ≥ 0 is a d-dimensional Brownian motionand
z ∈ R
d is a unit vector under the (cid:96) 2 norm, then the process W t :=
(cid:104) z, B t (cid:105) is a standard Brownianmotion.
Noting that g ( x (cid:48) )
=
f ( x (cid:48) ) − f ( x ) and that ( − W t ) t ≥ 0 is also a Brownian motion furnishesthe result.
We now use the characterization of privacy loss in Theorem 3.4 alongside the time-uniform concentration results for continuous time martingales found in Appendix A to construct two general familiesof privacy boundaries.
We now prove Theorem 3.6.
Proof of Theorem 3.6 .
Recall from Theorem 3.4 that we have the following bound
[Equation]
where, for a random variable X , the random variable X + is given by max( X, 0) .
First, by leveraging
Lemma A.3, we see that, with probability at least 1 − δ , for all n ∈ N , we have
[Equation]
proving that ψ Mρ is a valid δ -privacy boundary.
Likewise, by Lemma A.2, we have that
[Equation]
showing ψ La,b is a valid δ -privacy boundary.
C Proofs From Section
In this appendix, we provide proofs of the results in Section 5.
We start by proving the privacyguarantees for ReducedAboveThreshold .
Proof of Theorem 5.1 .
For ReducedAboveThreshold as described in Algorithm 1, on the event{ N ( x ) = n } , all information leaked about the underlying private dataset is contained in Alg 1: n ( x )and α 1: n ( x ) , where α n ( x ) is deﬁned to be the n th bit output by ReducedAboveThreshold .
Forany y ∈ X , let q y 1: n denote the joint density of (Alg 1: n ( y ) , α 1: n ( y )) , p y 1: n the marginal density of
Alg 1: n ( y ) , and p y 1: n ( · | · ) the conditional pmf of α 1: n ( y ) given the observed values of Alg 1: n ( y ) .
As such, for any neighboring datasets x ∼ x (cid:48) , on the event { N ( x ) = n } , the privacy loss of
ReducedAboveThreshold , denoted by L RAT ( x, x (cid:48) ) , is given by
[Equation]
where 0 n − 1 1 denotes the string of n − 1 0’s followed by a single 1.
In the last line we leverage thedeﬁnition of the privacy loss between Alg 1: n ( x ) and Alg 1: n ( x (cid:48) ) and deﬁne
[Equation]
Now, to ﬁnish the result, it sufﬁces to prove that, for any n , L n ( x, x (cid:48) )
≤ E n (Alg 1: n − 1 ( x )) .
Without loss of generality, we can assume all thresholds take the same value τ across rounds, aswe can always deﬁne the shifted function u (cid:48) n (Alg 1: n ( x ) , x ) :
= u n (Alg 1: n ( x ) , x ) − τ n + τ .
Toprove our desired inequality, we proceed largely in the same way as the proof of AboveThresholdfound in Lyu et al.
[2017], noting that conditioning on Alg 1: n ( x ) serves to ﬁx the utility functionsu 1 (Alg 1 ( x ) , · ) , . .
. , u n (Alg 1: n ( x ) , · ) and the privacy levels E 1 , E 2 (Alg 1 ( x )) , . .
, E n (Alg 1: n − 1 ( x )) .
For simplicity, going forward, we refer to the former quantities as u 1 ( · ) , . . .
, u n ( · ) and the latterquantities just as (cid:15) 1 , . .
, (cid:15) n .
The only remaining caveat that we must take care in handling variableamount of noise on the threshold introduced by LNR .
Going forward, let P 1: n denote the conditionalprobability P ( · | Alg 1: n ( x )) .
First, observe that we can write the numerator of L n ( x, x (cid:48) )
asp {+x (cid:0) 
 n −+}
[Equation]
1 | Alg 1: n ( x ) (cid:1) = (cid:90)
[Equation]
g [Equation] ( s [Equation]
. . . , s n )
[Equation]
i = [Equation] where g τ 1: n represents the density for the joint distribution of ( τ + Z (2∆ /(cid:15) m ))
nm =1 , where ( Z ( t )) t ≥ ηis as deﬁned in Equation (4).
We now need three inequalities.
The ﬁrst two are standard from theanalysis of Lyu et al.
[2017], so we do not provide a proof.
The third inequality is a product of ournovel ReducedAboveThreshold mechanism, and hence we provide a proof.
The inequalities are asfollows:
For i < n and ﬁxed s i , P 1: n ( u i ( x ) + ξ i < s i ) ≤ P 1: n ( u i ( x (cid:48) )
+ ξ i < s i + ∆) ,
for i = n and any s n , P 1: n ( u n ( x ) + ξ n ≥ s n ) ≤ e (cid:15) n / 2 P 1: n ( u n ( x (cid:48) )
+ ξ n ≥ s n + ∆) , and
for any s 1: n ∈ R n , g τ 1: n ( s 1 , . . .
, s n ) ≤ e (cid:15) n / 2 g τ 1:
n ( s 1 + ∆ , . . .
, s n + ∆) .
We now prove the third inequality.
We have that
[Equation]
where the ﬁrst equality follows from applying Bayes rule to the joint densities of the noisy thresholds,and the second equality follows from the fact that ( Z ( t )) forms a Markov process.
This in particularimplies that the density conditional density given the n th threshold satisﬁes g a 1: n − 1 ( s 1 , . . .
, s n − 1 |s n ) =
g b 1: n − 1 ( s 1 , . . .
, s n − 1 | s n ) for all a, b ∈ R .
The last inequality follows from examining theratio of densities of Lap( τ, 2∆ /(cid:15) n ) and Lap( τ − ∆ , 2∆ /(cid:15) n ) random variables.
Now, observe thatby a simple shift of parameters we have
[Equation]
Plugging this in, we have
[Equation]
Rearranging furnishes the desired result
[Equation]
We can also prove a corresponding utility guarantee for ReducedAboveThreshold .
As mentionedearlier, this utility guarantee is naive in the sense that it is derived from a union bound.
Thus,instead of plotting the utility guarantee in our experiments in Section 6, we instead plot empiricallyobserved loss/accuracy.
Additionally, for the utility guarantee to hold, the sequence of privacyfunctions ( E n ) n ≥ 1 must be constant functions, i.e. E n = (cid:15) n for each n .
We now state the formal,high-probability utility guarantee in the following proposition.
Proposition C.1.
Let ( p n ) n ≥ 1 be a sequence of non-negative numbers
such that (cid:80) ∞
i =1 p i = 1 , andlet γ ∈ (0 , 1) be a conﬁdence parameter.
Deﬁne the sequence of parameters ( η n ) n ≥ 1 by
[Equation]
Then, if N ( x ) is the time deﬁne in Theorem 5.1, with probability at least 1 − γ , we have
[Equation]
Proof.
The above utility guarantee follows from applying two simple union bounds.
First, we have
P   (cid:91) n
[Equation]
Second, we have that
P
[Equation]
η n / 2) = (cid:88) n ≥exp (cid:18) − (cid:15) n η n 2∆ (cid:19) ≤ γ (cid:88) n ≥p n = 1 .
Thus, with probability at least 1 − γ , we have simultaneously for all n ≥ 1 that | ξ n | ≤ η n / 2 and| ζ n | ≤ η n / 2 .
Thus, with the same probability, on round N ( x ) , we haveu N ( x ) ( x ) ≥ τ N ( x ) − η N ( x ) .
D Proofs From Section
We ﬁrst prove that the process deﬁned in Equation (4) has Laplace marginal distributions.
In whatfollows, we sometimes use the notation Z ( t ) interchangeably with Z t for convenience.
Theorem D.1.
Let ( Z t ) t ≥ η be the process deﬁend in Equation (4) .
Then, for any t ≥ η , we have
[Equation]
Proof.
Recall that if X ∼ Lap( s ) , then X has characteristic function ϕ s given by
[Equation]
Let φ denote the characteristic function of Z t − Z η .
Since Z η and Z t − Z η are independent, to show
Z t ∼ Lap( t ) , it sufﬁces to show that
[Equation]
Now, observe that the inhomogenous Poisson process ( P t ) t ≥ η can be written as ( (cid:101) P ( e t/ 2 ))
t ≥ log( η 2 )where (cid:101) P is a homogeneous Poisson process with rate λ = 1 on [log( η 2 ) , ∞ ) .
In terms of the process(cid:101) P , we can consider the process ( (cid:101) Z t ) t ≥ log( η 2 ) given by
[Equation]
where (cid:101) T n :
= inf { t ≥ log( η 2 ) : (cid:101) P t ≥ n } and (cid:101) T 0 = log( η 2 ) .
It is easy to see that (cid:101)
Z (log( t 2 ))
−(cid:101) Z (log( η 2 ))
=
d Z t − Z η .
Leveraging this identity, it follows that we haveφ
[Equation]
In the above, f ( n ) ( u 1 , . . .
, u n ) :
= n !
[log( t 2 /η 2 )] n is the distribution of the order statistics( U (1) , . .
, U ( n ) ) of n i.i.d.
random variables that are uniform on [log( η 2 ) , log( t 2 )] .
Essentially,what we have done is ﬁrst conditioned of the number of Poisson arrivals that occur in the interval[log( η 2 ) , log( t 2 )] .
Then, on the event { N ( t ) = n } , we condition again on the location of the narrivals, which we know to be uniformly distributed across the time interval.
Once the arrival locations are known, we can compute the conditional characteristic function, which is the the product ofcharacteristic functions as illustrated in the integral above.
Now, we show inductively that
[Equation]
1 + λ 2 t(cid:19)(cid:21) n .
The base case of n = 1 is trivially true.
Now, we have that
[Equation]
Leveraging this identity and picking up from the expression for φ ( λ ) in Equation (5)
[Equation]
we have that
[Equation]
This proves the desired result.
The above proof can also be leveraged to show
that, for any ﬁnite ﬁxed sequence of times ( t n ) n ∈
[ K ] ,( Z ( t 1 ) , . . .
, Z ( t K )) has the same distribution as ( ζ 1 , . . .
, ζ K ) , where ( ζ n ) n ∈ [ K ] is the Laplace processassociated with times ( t n ) n ∈
[ K ] as outlined in Equation (3).
This justiﬁes that the process ( Z ( t ))
t ≥ ηis in fact a continuous time generalization of the aforementioned discrete time process.
E Skellam Noise Reduction
Lastly, we brieﬂy discuss how to generate a noise reduction mechanism for Skellam noise [Agarwalet al., 2021].
Recall that a random variable X has a Skellam distribution with parameters λ 1 and λ 2if X =
d Y 1 − Y 2 , where Y 1 ∼ Poisson( λ 1 ) and Y 2 ∼ Poisson( λ 2 ) are independent Laplace randomvariables.
For succinctness, we write X ∼ Skell ( λ 1 , λ 2 ) .
Let ( P 1 ( t ))
t ≥ 0 and ( P 2 ( t ))
t ≥ 0 be two independent, homogeneous Poisson process with rates λ 1 andλ 2 respectively.
Observe that the continuous time process ( X t ) t ≥ 0 given by X t := P 1 ( t ) − P 2 ( t ) isclearly Markovian, has independent increments, and has X t ∼ Skell( tλ 1 , tλ 2 ) .
Thus, ( X t ) t ≥ 0 canbe used to deﬁne a Skellam noise reduction mechanism by releasing ( f ( x ) + X T n ( x ) ) n ≥ 1 for somesequence of time functions ( T n ) n ≥ 1 . F Additional Experimental Details
Parameter settings: We set the regularization parameter to be λ = 0 . 05 and note that the (cid:96) 2 and(cid:96) 1 -sensitivity for the output perturbation of logistic regression are respectively 2 nλ and 2 √ d nλ .
Likewise,for covariance perturbation in ridge regression, the (cid:96) 2 -sensitivities for privately releasing X T X and
{+X T y are both 2 . 0 , and the corresponding (cid:96) 1 -sensitivities for releasing these quantities are 2 . 
+}
{+d and2 . 
+}
√ d respectively [Ligett et al., 2017, Chaudhuri et al., 2011].
We set the failure probability for BMto be δ = 10 − 6 , and in each task map privacy parameters ( (cid:15) n ) to times ( t n ) using the linear privacyboundary ψ La,b optimized for tightness at (cid:15) = 0 . 3 .
Optimizing privacy boundaries: We provide a high level description of how one may set theparameters associated with the privacy boundaries discussed in Theorem 3.6.
Let us consider thecase of the mixture boundary ψ Mρ for illustrative purposes.
Suppose a data analyst desires that the ﬁnal level of privacy loss obtained by interacting with the
Brownian mechanism should be approximately (cid:15) .
Then, intuitively, the analyst should want to addthe variance of the Gaussian noise added to be as small as possible when the privacy boundary takesvalue (cid:15) .
In mathematical notation, the analyst wants to ﬁnd a parameter ρ ∗ satisfying
[Equation]
where we note that the inverse function ( ψ Mρ ) − 1 exists as ψ Mρ is strictly increasing.
While this inversehas no closed form in general, the parameter ρ ∗ can be efﬁciently computed using several lines ofcode.
A similar, even more straightforward computation can be conducted for the linear privacyboundary.
Simulating Noise Reduction Mechanisms
We brieﬂy describe how a data analyst can produce samples from the Brownian mechanism and the Laplace noise reduction mechanism.
First, since T 1 ( x ) is a constant, wehave BM 1 ( x ) ∼
N
[Equation]
f ( x ) , T 1 ( x )) .
Then, given BM 1: m − 1 ( x ) , we have BM m ( x ) ∼
[Equation]
(cid:17) .
Since simulating the Brownianmechnaism only requires normal samples, it can be efﬁciently computed.
Second, to sample from LNR , one can ﬁrst generate the the points of arrival of the inhomogeneous
Poisson process ( P t ) t ≥ η up to time T 1 ( x ) .
Let T 1 , . .
, T N denote these arrival times, where wenote that N , the number of arrivals up to time T 1 ( x ) , is a random variable.
Then, one can generate
Y m ∼ Lap( T m ) for m ≤ N .
From this information, the process ( Z t ) η ≤ t ≤ T 1 ( x ) can be readilycomputed, as in Equation (4).