F ORGET U NLEARNING : T OWARDS T RUE D ATA -D ELETION IN M ACHINE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Unlearning has emerged as a technique to efﬁciently erase information of deleted records from learned models.
We show, however, that the inﬂuence created by the original presence of a data point in the training set can still be detected after running certiﬁed unlearning algorithms (which can result in its reconstruction by an adversary).
Thus, under realistic assumptions about the dynamics of model releases over time and in the presence of adaptive adversaries, we show that unlearning is not equivalent to data deletion and does not guarantee the “right to be forgotten.”
We then propose a more robust data-deletion guarantee and show that it is necessary to satisfy differential privacy to ensure true data deletion.
Under our notion, we propose an accurate, computationally efﬁcient, and secure data-deletion machine learning algorithm in the online setting based on noisy gradient descent algorithm.
I NTRODUCTION
Many corporations today collect their customers’ private information to train Machine Learning (ML) models that power a variety of services, encompassing recommendations, searches, targeted ads, and more.
To prevent any unintended use of personal data, privacy policies, such as the General Data
Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), require that these corporations provide the “ right to be forgotten ” (RTBF) to their data subjects—if a user wishes to revoke access to their data, an organization must comply by erasing all information about the user without undue delay (which is typically a month).
This includes ML models trained in standard ways as model inversion (Fredrikson et al., 2015) and membership inference attacks (Shokri et al., 2017;
Carlini et al., 2019) demonstrate that individual training data can be exﬁltrated from these models.
Periodic retraining of models after excluding deleted users can be costly.
So, there is a growing interest in designing computationally cheap Machine Unlearning algorithms as an alternative to retraining for erasing the inﬂuence of deleted data from (and registering the inﬂuence of added data to) trained models.
Since it is generally difﬁcult to tell how a speciﬁc data point affects a model,
Ginart et al.
(2019) propose quantifying the worst-case information leakage from an unlearned model through an unlearning guarantee on the mechanism, deﬁned as a differential privacy (DP) like ( ε, δ ) -indistinguishability between its output and that of retraining on the updated database.
With some minor variations in this deﬁnition, several mechanisms have been proposed and certiﬁed as unlearning algorithms in literature (Ginart et al., 2019; Izzo et al., 2021; Sekhari et al., 2021; Neel et al., 2021;
Guo et al., 2019; Ullah et al., 2021).
However, is indistinguishability to retraining a sufﬁcient guarantee of data deletion?
We argue that it is not.
In the real world, a user’s decision to remove his information is often affected by what a deployed model reveals about him.
The same revealed information may also affect other users’ decisions.
Such adaptive requests make the records in a database interdependent, causing a retrained model to contain inﬂuences of a record even if the record is no longer in the training set.
We demonstrate on a certiﬁed unlearning mechanism that if an adversary is allowed to design an adaptive requester that interactively generates database edit requests as a function of published models, she can re-encode a target record in the curator’s database before its deletion.
We argue that under adaptive requests, measuring data-deletion via indistinguishability to retraining (as proposed by Gupta et al.
(2021)) is fundamentally ﬂawed because it does not capture the inﬂuence a record might have previously had on the rest of the database.
Our example shows a clear violation of the RTBF since even after retraining on the database with the original record removed, a model can reveal substantial information about the deleted record due to the possibility of re-encodings.
Is an unlearning guarantee a sound and complete measure of data deletion when requests are non-adaptive?
Again, we argue that it is neither.
A sound data-deletion guarantee must ensure the non-recovery of deleted records from an inﬁnite number of model releases after deletion.
However, approximate indistinguishability to retraining implies an inability to accurately recover deleted data from a singular unlearned model only, which we argue is not sufﬁcient.
We show that certain algorithms can satisfy an unlearning guarantee yet blatantly reveal the deleted data eventually over multiple releases.
The vulnerability arises in algorithms that maintain partial computations in internal data structures for speeding up subsequent deletions.
These internal states can retain information even after record deletion and inﬂuence multiple future releases, making the myopic unlearning guarantee unreliable in an online setting.
Several proposed unlearning algorithms in literature (Ginart et al., 2019; Neel et al., 2021) are stateful (rely on internal states) and, therefore, cannot be trusted.
Secondly, unlearning is an incomplete notion of data deletion as it excludes valid data-deletion mechanisms that do not imitate retraining.
For instance, a (useless) mechanism that outputs a ﬁxed untrained model on any request is a valid deletion algorithm.
However, since its output is easily distinguishable from retraining, it fails to satisfy any meaningful unlearning guarantees.
This paper proposes a sound deﬁnition of data deletion that does not suffer from the abovementioned shortcomings.
According to our notion, a data-deletion mechanism is reliable if A) it is stateless (i.e., it maintains no internal data structures), and B) generates models that are indistinguishable from some random variable that is independent of the deleted records.
Statelessness thwarts the danger of sustained information leakage through internal data structures after deletion.
Moreover, by deﬁning data deletion as indistinguishability with any deleted-record independent random variable as oppsed to the output of retraining, we ensure reliability in presence of adaptive requests that create dependence between current and deleted records in the database.
In general, we show that data-deletion mechanisms must be differentially private with respect to the remaining records to be reliable when requests are adaptive.
DP also protects against membership inference attacks that extract deleted records by looking at models before and after deletion (Chen et al., 2021).
We emphasize that we are not advocating for doing data deletion through differentiallyprivate mechanisms simply because it caps the information content of all records equally, deleted or otherwise.
Instead, a data-deletion mechanisms should provide two differing information reattainment bounds; one for records currently in the database in the form of a differential privacy guarantee and the other for records previously deleted in the form of a data-deletion guarantee.
We also provide a reduction theorem that if a mechanism is differentially private with respect to the remaining records and satisﬁes a data-deletion guarantee under non-adaptive edit requests, then it also satisﬁes a data-deletion guarantee under adaptive requests.
Based on this reduction, we redeﬁne the problem of data-deletion as designing a mechanism that (1.) satisﬁes a data-deletion guarantee against nonadaptive deletion requests, (2.) is differentially private for remaining records, and (3.) has the same utility guarantee as retraining under identical differential privacy constraints.
We judge the usefulness of a data-deletion mechanism based on its computational savings over retraining.
For our reﬁned problem formulation, we provide a data-deletion solution based on Noisy Gradient Descent (Noisy-GD), a popular differentially private learning algorithm
(Bassily et al., 2014; Abadi et al., 2016; Chourasia et al., 2021).
Our solution demonstrates a powerful synergy between data deletion and differential privacy as the same noise needed for the privacy of records in the database also rapidly erases information regarding records deleted from the database.
We provide a datadeletion guarantee for Noisy-GD in terms of Rényi divergence (Rényi et al., 1961) bound (which implies ( ε, δ ) -indistinguishability
(Mironov, 2017)).
For convex and smooth losses, we certify that under a ( q, ε dd ) -Rényi data-deletion and ( q, ε dp ) -Rényi DP constraint, our Noisy-GD based deletion mechanism for d -dimensional models over n -sized databases under adaptive edit requests that modify no more than r records can maintain optimal excess empirical risk of the order O (cid:0) qd ε dp n 2 (cid:1) while saving Ω( n log(min { n r , n (cid:113) ε dd qd } ) computations in gradient complexity.
Our utility guarantee matches the known lower bound on private empirical risk minimization under same privacy budget (Bassily et al., 2014).
We also provide data-deletion guarantee in the non-convex setting under the assumption that loss function is bounded and smooth, and show a computational saving of Ω( dn log nr ) in gradient complexity while maintaining an excess risk of ˜ O (cid:0) qd ε dp n 2 + 1 n (cid:113) q ε dp (cid:1) .
M ODEL AND P RELIMINARIES 2. I NDISTINGUISHABILITY N OTIONS AND D IFFERENTIAL P RIVACY
We provide the basics of indistinguishability of random variables (with more details in Appendix D).
Let Θ , Θ (cid:48) be two random variables in space O with probability densities ν , ν (cid:48) respectively.
Deﬁnition 2.1 ( ( ε, δ ) -indistinguishability (Dwork et al., 2014)) .
We say Θ and Θ (cid:48) are ( ε, δ ) - indistinguishable and write Θ ε,δ ≈ Θ (cid:48)
if, for all O ⊂ O ,
[Equation]
O ]
[Equation]
and P [Θ (cid:48)
∈ O ]
≤ e ε P [Θ ∈ O ] + δ.
Deﬁnition 2.2 (Rényi divergence (Rényi et al., 1961)) .
Rényi divergence of ν w.r.t.
ν (cid:48) of order q >is deﬁned as
[Equation]
where E [Equation] ( ν (cid:107) [Equation]
=
E
[Equation]
when ν is absolutely continuous w.r.t.
ν (cid:48) (denoted as ν (cid:28) ν (cid:48) ).
If ν (cid:54)(cid:28) ν (cid:48) , we’ll say R q ( ν (cid:107) ν (cid:48) )
= ∞ .
Remark 1.
Rényi divergence is assymetric (i.e. R q ( ν (cid:107) ν (cid:48) )
(cid:54) =
R q ( ν (cid:48)
(cid:107) ν ) ) and implies indistinguishability only in one direction.
Mironov (2017, Proposition 3) show that R q ( ν (cid:107) ν (cid:48) ) ≤ ε 0 implies
P [Θ ∈ O ] ≤ e ε P [Θ (cid:48) ∈
O ]
+ δ with ε = ε
+ log1 /δ
q − 1 and any 0 < δ < 1 .
Deﬁnition 2.3 (Differential Privacy (Dwork et al., 2014; Mironov, 2017)) . A randomized mechanism
M : X n → O is ( ε, δ ) - differentially private if M ( D ) ε,δ ≈ M ( D (cid:48) ) for all neighbouring databases D , D (cid:48) ∈
X n . Similarly, M is ( q, ε ) - Rényi differentially private if R q ( M ( D ) (cid:107)M ( D (cid:48) )) ≤ ε . 2. L EARNING F RAMEWORK : ERM
Let D be a database of n ordered records taken from a data universe X
and let O be the space of learnable parameters and any associated auxiliary metadata (what constitutes a metadata is clariﬁed later). Let (cid:96) ( θ ; x ) : O × X
→
R be a loss function of a parameter θ ∈ O for a record x ∈ X . In this paper, we consider the problem of empirical risk minimization (ERM) of the average (cid:96) ( θ ; x ) over records in the database D under L 2 regularization r ( θ ) = λ (cid:107) θ (cid:107) 22 2 , i.e., the minimization objective is
L D ( θ ) = 1 n (cid:80) x ∈D (cid:96) ( θ ; x ) + r ( θ ) . The excess empirical risk of a model Θ for D by is deﬁned as err(Θ; D )
=
E [ L D (Θ) − L D ( θ ∗D )] , where θ ∗D = arg min θ ∈O L D ( θ ) and expectation is taken over Θ .
We build on a popular DP-ERM algorithm called Noisy-GD (Abadi et al., 2016), described in Algorithm 1 below, and provide Rényi DP guarantees on it in Appendix G.3.
Algorithm
Require: Database D ∈ X n , start model Θ 0 ∈
O , number of steps K ∈ N .
K do 2:
[Equation]
[Equation]
4: Output
2. O NLINE E DIT R EQUESTS AND M ACHINE U NLEARNING
Suppose that any database D ∈ X n can be modiﬁed by edit requests that replaces r distinct records 1 .
Deﬁnition 2.4 (Edit request) .
A replacement operation (cid:104) ind , y (cid:105) ∈ [ n ] × X on a database
D = ( x 1 , · · · , x n ) performs the following modiﬁcation:
[Equation]
Let r ≤ n and U r = [ n ] r (cid:54) =
× X r .
An edit request u = {(cid:104) ind 1 , y 1 (cid:105) , · · · , (cid:104) ind r , y r (cid:105)} ∈ U r on D is deﬁned as batch of r replacement operations modifying distinct indices atomically, i.e.
[Equation]
where ind i (cid:54) = ind j for all i (cid:54) =
j.
[Equation]
Let O denote the space of model parameters plus any state variables or data-structures that may be leveraged for processing edit requests and Φ be the space of publishable outcomes such as sanitized models or predictions. For an initial database D 0 ∈ X n and an unbounded sequence of edit requests ( u i ) i ≥ 1 , the job of a data curator is to train a model ˆΘ i ∈ O with small empirical risk for each database D i = D 0 ◦ u 1 ◦ · · · ◦ u i and use it to release a corresponding publishable outcome, φ i ∈ Φ . For this task, the trivial approach of retraining from scractch by executing a learning algorithm on each D i could be computationally expensive.
This initiated the study of machine unlearning algorithms (Cao & Yang, 2015; Ginart et al., 2019; Guo et al., 2019; Izzo et al., 2021) that avoid the cost of retraining by instead ﬁne-tuning an already trained model ˆΘ i − 1 to cheaply erase (introduce) the inﬂuence of deleted (added) records in edit request u i to D i − 1 for producing the next model ˆΘ i .
In this paper, we adopt the machine unlearning formulation of Gupta et al.
and Neel et al.
(2021) described as follows. The curator comprises of three mechanisms: a learning algorithm A : X n → O , a data-deletion or unlearning algorithm ¯A : X n × U r ×
O
→ O , and a publish function f pub : O → Φ . To generate the initial model with accompanying state metadata ˆΘ 0 ∈ O on the initial database D 0 ∈ X n , the curator executes the learning algorithm A( D 0 ) .
Thereafter, to process an incoming edit request u
i ∈
U r at any step i ≥ 1 , the curator executes data-deletion algorithm ¯ A( D i − 1 , u i , ˆ Θ i − 1 ) that maps the current database
D
i − 1 , the edit request u i , and the current model with metadata ˆΘ i − 1 to the next model with metadata ˆΘ i ∈ O .
While all the generated models and corresponding metadata is kept secret, the curator uses the publish function to generate the publishable outcome φ i = f pub (ˆΘ i ) at every step i ≥ 0 . Gupta et al.
(2021) note that edit requests in real world could often be adaptive , i.e., a request u i may depend on (a subset of) the history of prior published outcomes φ ≤ i = ( φ 1 , · · · , φ i ) . For instance, a voter may decide to change his inclination after seeing pre-election results. They model such an intearctive environment through an adaptive requester deﬁned as follows.
Deﬁnition 2.5 (Update requester (Gupta et al., 2021)) .
A p -adaptive r -requester is a mapping
Q : Φ ≤ p × U r ∗ → U r that takes as input a maximum of p of the published outcomes generated by the curator at arbitrary edit steps s 1 < s 2 < · · · < s p , and the entire history of previously generated edit requests to generate the next edit request.
For a p -adaptive r -requester Q , the edit request u i at any step i ≥ 1 can be written as
[Equation]
such that s j < i .
We refer to 0 -adaptive requesters as non-adaptive.
And, by ∞ -adaptive requesters, we mean requesters that have access to the entire history of interaction transcript ( φ <i ; u <i ) .
Since, unlike retraining, an unlearning algorithm ﬁne-tunes a model containing the information of records to be deleted, we need statistical guarantees on the worst-case amount of information that might still remain in the unlearned model.
Ginart et al.
(2019) and Guo et al. (2019) propose quantifying data-deletion ability of an algorithm ¯ A based on its ( ε, δ ) -indistinguishability w.r.t.
the fresh-retraining algorithm A , calling it an unlearning guarantee (more details in Appendix E.1).
In this paper, we mainly consider the extension of unlearning deﬁnitions by Neel et al. (2021) and Gupta et al.
(2021) to the online setting of arbitrarily long and adaptive edit sequences 2 .
Deﬁnition 2.6 ( ( ε, δ ) -unlearning
(Neel et al., 2021; Gupta et al., 2021)) .
We say that ¯A is an ( ε, δ ) - unlearning algorithm for A under a publish function f pub , if for all initial databases D 0 ∈ X n and all non-adaptive 1 -requesters Q , the following condition holds.
For every edit step i ≥ 1 , and for all generated edit sequences u
[Equation]
i
[Equation]
u
[Equation] · · · , u i ) , [Equation] If the same condition holds for all ∞ -adaptive 1 -requesters Q , we say that ¯A is an ( ε, δ ) - adaptiveunlearning algorithm for A
E XISTING U NLEARNING G UARANTEES A RE U NSOUND AND I NCOMPLETE
Data deletion under the law of “right to be forgotten” (RTBF) is an obligation to permanently erase all information about an individual upon a veriﬁed request.
In order to comply, a corporation’s actions must not reveal any information identiﬁable or linkable to a deleted user in the future.
In this section, we argue that unlearning guarantee in Deﬁnition 2.6 is neither a sound nor a complete measure of data-deletion from ML models that RTBF enforces. Threat model.
Suppose, for an arbitrary step i ≥ 1 , an adversary is interested in ﬁnding out the identity of a record in the database D
i − 1 that was deleted by the edit request u i .
Since RTBF is violated only when the curator reveals information after the deletion request , we assume that the adversary only has access to the post-deletion releases by the curator, i.e., she observes φ i , φ i +1 , · · · .
Additionally, we also assume that the adversary knows how users might react to a published outcome.
That is to say, our adversary knows some dependence relationship between random variables φ 0 , · · · , φ i − 1 and u 1 , · · · , u i − 1 , but does not explicitly observe these random variables. For instance, an adversary might know that if the outcome φ 1 predicts that "Donald Trump is winning the election," then some democratic users might delete their data while some new republican users might contribute their data to the curator. So, even though the adversary does not observe the actual outcome φ 1 or the ensuing edit request u 2 , and so on, she can still exploit knowledge about this dependence a-posteriori to infer the identity of a deleted record. To capture the worst-case knowledge about the dependence between unobserved outcomes and unobserved edit requests, we model our adversary to have the power to design an adaptive requester Q that interacts with the curator in the ﬁrst i − 1 steps.
However, the adversary does not observe the interaction transcript ( φ 0 , u 1 , φ 1 , · · · u i − 1 , φ i − 1 ) of Q .
Unsoundness due to secret states. The unlearning Deﬁnition 2.6 is a bound on information leakage about a deleted record through a single released outcome .
However, our adversary can observe multiple (potentially inﬁnite) releases after deletion.
We argue that algorithms satisfying Deﬁnition 2. can lead to blatant non-privacy of a deleted record under our threat model, even for a weaker adversary that cannot design an adaptive requester (i.e., knows nothing about the dependence between unobserved outcomes and edit requests).
The vulnerability arises as Deﬁnition 2.6 permits algorithms to maintain secret states while using a publishing function f pub for releases.
These internal states may propagate encoded information about records even after their deletion from the database.
So, every subsequent release by an unlearning algorithm can reveal new information about a record that was purportedly erased multiple edits earlier.
We demonstrate in the following theorem that a certiﬁed unlearning algorithm can reveal a limited amount of information about a deleted record per release so as not to break the unlearning certiﬁcation, yet eventually reveal everything about the record to an adversary that observes enough future releases.
Theorem 1.
For every ε > 0 , there exists a pair (A , ¯A) of algorithms that satisfy ( ε, 0) -unlearning under a publish function
f pub such that for all non-adaptive 1 -requesters Q , their exists an adversary that can correctly infer the identity of a record deleted at any arbitrary edit step i ≥ 1 by observing only the post-edit releases φ ≥ i =
( φ i , φ i +1 , · · · ) .
Unsoundness due to adaptivity. For an adversary that knows some dependence relationship between the unobserved outcomes and edit requests, a much more severe violation of RTBF may occur, even when an unlearning algorithm does not maintain secret internal states and perfectly imitates retraining (i.e., satisfy a (0 , 0)
-adaptive unlearning guarantee for identity publish function f pub ( θ ) = θ ).
This vulnerability arises because the indistinguishability in Deﬁnition 2.6 protects the privacy of deleted records but not that of records currently present.
A certiﬁed adaptive unlearning algorithm is allowed to reveal unbounded information about a target record before its deletion. This revealed information can have a major inﬂuence on the subsequent edit requests in the worst case, potentially causing the curator’s database to have patterns speciﬁc to the identity of the target record even after its deletion. An adversary knowing the possible patterns and their causes (i.e., the dependence relationship between unobserved outcomes and requests) can therefore infer the target records’s identity from post-deletion releases.
We concretize this vulnerability in the following theorem.
Theorem 2.
There exists a pair (A , ¯A) of algorithms that satisfy (0 , 0) -adaptive-unlearning for an identity publish function f pub ( θ ) = θ such that by designing a 1 -adaptive 1 -requester Q , an adversary, even with no access to Q ’s interaction transcript, can infer the identity of a record deleted at any arbitrary edit step i > 3 with probability at-least 1 − (1 / 2) i − 3 from the post-edit release φ i .
Incompleteness.
Another issue with unlearning guarantees is that valid data-deletion algorithms may fail to satisfy it.
As per Deﬁnition 2.6, the publishable outcome generated by a data-deletion mechanism must imitate that of a retraining algorithm for satisfying an unlearning guarantee.
However, imitating retraining is not necessary for data deletion.
For instance, consider a (useless) mechanism ¯A that outputs a ﬁxed untrained model in O regardless of its inputs.
This ¯A would be an unacceptable unlearning algorithm for most re-training algorithms
A under an identity publish function f pub ( θ ) = θ .
However, f pub (¯A( · )) contains no information about the deleted records and should be acceptable under the data-deletion paradigm. Remark 2.
Several prior works on machine unlearning propose data-deletion deﬁnitions similar to Deﬁnition 2.6 but without an explicit f pub and for the ofﬂine setting of a single-stage deletion to remove r many records from a learned model (Ginart et al., 2019; Guo et al., 2019; Sekhari et al., 2021).
In Appendix E.1, we show that these ofﬂine unlearning deﬁnitions are also unreliable when deletion requests are adaptive.
In light of our demonstration of unsoundness of these data-deletion guarantees, we remark that several certiﬁed unlearning algorithms in literature should not be trusted to ensure RTBF.
R EDEFINING D ATA -D ELETION IN M ACHINE L EARNING
In this section, we redeﬁne data deletion in Machine Learning to address the problems with the notion of unlearning that we demonstrate in the preceding section.
The ﬁrst change we propose is to rule out the possibility of information leakage through internal data structures (as shown in Theorem 1) by requiring deletion mechanisms to be stateless .
That is, the models produced by learning or the data-deletion algorithm are directly released without applying any publish function f pub .
Secondly, the following deﬁnition of a data-deletion guarantee ﬁxes the security blind spot of an adaptive unlearning guarantee.
As demonstrated in Theorem 2, an adaptive requester can encode patterns speciﬁc to a target record in the database by making edit decisions in response to the observed outcomes.
Thus, being indistinguishable from retraining on the edited database does not guarantee data deletion, as the target’s information remains extractable even after the target record
’s deletion, potentially revealing its identity.
In our deﬁnition, we account for an adaptive adversary’s inﬂuence by measuring the indistinguishability of a data-deletion mechanism’s output from some random variable that is independent of the deleted record.
Deﬁnition 4.1 ( ( q, ε ) -data-deletion under p -adaptive r -requesters) .
Let q > 1 , ε ≥ 0 , and p, r ∈ N .
We say that an algorithm pair (A , ¯A) satisﬁes ( q, ε ) -data-deletion under p -adaptive r -requesters if the following condition holds for all p -adaptive r -requester Q .
For every step i ≥ 1 , there exists a randomized mapping π Q i : X n → O such that for all initial databases D 0 ∈ X n ,
[Equation]
We argue that the above deﬁnition is a sound guarantee on data-deletion. Suppose that an adversary is interested in identifying a record at index ‘ ind ’ in D 0 that is being replaced with record ‘ y ’ by one of the replacement operations in edit request u i ∈
U r . The inequality (7) above implies that even with the power of designing an adaptive requester Q , no adversary observing the unlearned model ¯A( D i − 1 , u i , ˆΘ i − 1 ) can be too conﬁdent that the observation was not from π Q i ( D 0 ◦ (cid:104) ind , y (cid:105) ) , a distribution that contains no information about the target record {+D 
+}
[ind] by construction. More formally, we provide the following soundness guarantee for Deﬁnition 4.1. Theorem 3 (Data-deletion Deﬁnition 4.1 is sound) .
If the algorithm pair (A , ¯A) satisﬁes ( q, ε ) - data-deletion guarantee under all p -adaptive r -requesters, then even with the power of designing an p -adaptive r -requester Q that interacts with the curator before deletion of a target record at any step i ≥ 1 , any adversary observing only the post-deletion releases (ˆΘ i , ˆΘ i +1 , · · · ) has its membership inference advantage for inferring a deleted target bounded as
[Equation] Note that the bound in (8) approaches 0 as q → ∞ and ε → 0 , implying Deﬁnition 4.1 is sound.
Remark 3.
A non-adaptive requester Q is equivalent to ﬁxing the request sequence ( u i ) i ≥ 1 a-priori. Hence, given a non-adaptive Q , the database D i ◦ (cid:104) ind , y (cid:105) is a deterministic function of the database
D 0 ◦ (cid:104) ind , y (cid:105) for any i ≥ 1 , thanks to the commutativity of ‘ ◦ ’.
Since (cid:104) ind , y (cid:105) ∈ u i , we remark that for a non-adaptive requester Q , the random variable π Q
i ( D 0 ◦ (cid:104) ind , y (cid:105) ) in (7) can be the output π ( D i ) of any randomized map π : X n → O , including the learning algorithm A . Connection with DP.
We highlight that our data-deletion guarantee on pair (A , ¯A) is an information bound on the records that were deleted, while the standard DP guarantee on A and ¯A is an information bound on the records currently present in the database.
In the following theorem, we show that DP with respect to existing records is a necessary condition for (A , ¯A) to satisfy adaptive data-deletion. Theorem 4 (DP is necessary for adaptive data-deletion) . If learning algorithm A : X n → O is not (0 , δ ) -DP with respect to the replacement of a single record and deletion
algorithm ¯A : X n ×U×O → O is not (0 , δ ) -DP with respect to the replacement of a single record that is not being deleted, then the pair (A , ¯A) cannot satisfy ( q, δ 4 / 2) -data-deletion under 1 -adaptive 1 -requester for any q > 1 . Additionally, if A and ¯A does satisfy DP with respect to existing records then a data-deletion bound under non-adaptive requesters reduces to a data-deletion bound under adaptive requesters.
Theorem 5 (Non-adaptive data-deletion with Rényi DP implies adaptive data-deletion) .
If an algorithm pair (A , ¯A) satisﬁes ( q, ε dd ) -data-deletion under all non-adaptive r -requesters and ( q, ε dp ) - Rényi DP, then it also satisﬁes ( q, ε dd + pε dp ) -data-deletion under all p -adaptive r -requesters. We provide additional discussion on this reduction theorem in Appendix F.1.
By virtue of our reduction Theorem 5, we reformulate the data deletion problem in ML as follows.
Problem Deﬁnition.
Let constants q > 1 , 0 < ε dd ≤ ε dp , and α > 0 .
The goal is to design a learning mechanism A : X n → O and a deletion mechanisms ¯A : X n × U
r × O → O such that (1.)
both A and ¯A satisfy ( q, ε dp ) -Rényi DP with respect to records in the input database, (2.)
pair (A , ¯A) satisﬁes ( q, ε dd ) -data deletion guarantee for all non-adaptive r -requesters Q , ( and, all models (ˆΘ i )
i
≥ 0 produced by (A , ¯A , Q on any D ∈ X n have err(ˆΘ i ; D i ) ≤ α .
We judge the beneﬁt of data deletion based on the computation saving that ¯A offers over A per request.
That is, if we want err(¯Θ
i ; D i ) ≤ α for all i ≥ 0 , where ¯Θ i is trained from scratch by A( D i ) , how large is Cost (A) compared to Cost (¯A) .
Note that α should ideally be the optimal excess risk attainable under the ( q, ε dp ) -RDP constraint.
We remark that one may use any reasonable measure of utility in the third constraint, such as population risk instead of excess empirical risk.
D ATA D ELETION U SING N OISY G RADIENT D ESCENT
This section proposes a simple and effective data-deletion solution based on Noisy-GD Algorithm 1.
Our proposed approach falls under the Descent-to-Delete framework proposed by Neel et al. (2021), wherein, after each deletion request u i , we run Noisy-GD starting from the previous model ˆΘ i −and perform a small number of gradient descent steps over records in the modiﬁed database D i = D i − 1 ◦ u i ; sufﬁcient to ensure that the information about the deleted records is reduced within a desired bound in the subsequent model ˆΘ i .
Formally, our data-deletion solution is deﬁned as follows.
Deﬁnition 5.1 (Data-deletion based on Noisy-GD) .
Let space of model parameters be O = R d .
Our learning algorithm
A Noisy-GD ( D 0 ) =
Noisy-GD ( D 0 , Θ 0 , K A ) runs Noisy-GD algorithm for
K A iterations to generate the ﬁrst learned model ˆΘ 0 on the input database D 0 ∈ X n , with Θ 0 sampled from a weight initialization distribution ρ .
Our data-deletion algorithm ¯A Noisy-GD ( D i − 1 , u i , ˆΘ i − 1 ) =
Noisy-GD ( D i − 1 ◦ u i , ˆΘ i − 1 , K ¯A ) processes an edit request u
i ∈
U r by running K ¯A iterations of Noisy-GD algorithm on the updated database D i
=
D i − 1 ◦ u i to generate the updated model ˆΘ i from the current model ˆΘ i − 1 ∈ R d .
For this setup, our objective is to provide conditions under which the algorithm pair (A Noisy-GD , ¯A Noisy-GD ) satisﬁes objectives (1.) , (2.) , and (3.) as stated in the problem deﬁnition above and analyze the computational savings of using ¯A Noisy-GD over A Noisy-GD in terms of the gradient complexity n ( K A − K ¯A ) .
5. D ATA -D ELETION AND A CCURACY G UARANTEES U NDER C ONVEXITY
We give the following guarantee for pair (A Noisy-GD , ¯A Noisy-GD ) under convexity of loss (cid:96) ( θ ; x ) .
Theorem 6 (Accuracy, privacy, deletion, and computation tradeoffs) .
Let constants λ, β, L > 0 , q > 1 , and 0 < ε dd ≤ ε dp .
Deﬁne constant κ = λ + βλ .
Let the loss function (cid:96) ( θ ; x ) be twice differentiable, convex, L -Lipschitz, and β -smooth, the regularizer be r ( θ )
=
λ 2 (cid:107) θ (cid:107) 22 .
If the learning rate be η = 1 2( λ + β ) , the gradient noise variance is σ 2 = 4 qL 2 λε dp n 2 , and the weight initialization distribution is ρ = N (cid:16) 0 ,
[Equation]
(cid:17) , then
both A Noisy-GD and ¯A Noisy-GD are ( q, ε dp ) -RDP for any K A , K ¯A ≥ 0 , (2.)
pair (A Noisy-GD , ¯A Noisy-GD ) satisﬁes ( q, ε dd ) -data-deletion all non-adaptive r -requesters
[Equation]
) and all models in (ˆΘ i ) i ≥ 0 produced by (A Noisy-GD , ¯A Noisy-GD , Q ) on any D 0 ∈ X n , where
Q is any r -requester, have an excess empirical risk err(ˆΘ i ; D i ) =
O (cid:16) qd ε dp n 2 (cid:17) if K A ≥ 4 κ log (cid:18) ε dp n4 qd (cid:19) , and K ¯A ≥ 4 κ log max (cid:26)
[Equation]
qd (cid:27) .
Proof of Theorem 6 can be found in Appendix G.4.
Our utility upper bound for data-deletion matches the theoretical lower bound of Ω(min (cid:8) 1 , d ε 2 n 2 (cid:9) ) in Bassily et al.
(2014) for the best attainable utility of ( ε, δ ) -DP algorithms on Lipschitz, smooth, strongly-convex loss functions 3 .
Thus, our data-deletion algorithm, ¯A Noisy-GD , incurs no additional cost in utility but saves substantial computation costs.
Our data-deletion algorithm offers a computation saving of Ω( n log min { nr , n (cid:113)
ε dd qd } ) per request while guaranteeing privacy, deletion, and optimal utility and without unsafe internal data structures.
This saving is better than all existing unlearning algorithms in literature that we know of, and we present a detailed comparison in Table 1.
Also, observe that for satisfying ( q, ε dp ) -RDP and ( q, ε dd ) -data-deletion for non-adaptive rrequesters, the number of iterations K ¯A needed is independent of the size, r , of the deletion batch, depending solely on the ratio ε dd ε dp .
However, the number of iterations required for ensuring optimal utility with differential privacy grows with r .
As such, we highlight that when deletion batches are sufﬁciently large, i.e., r ≥ (cid:113) qdε dd , ensuring optimal utility under ( q, ε dp ) -RDP results in ( q, ε dd ) - data-deletion for free, thus demonstrating synergy between privacy and data deletion.
5. D ATA -D ELETION AND U TILITY G UARANTEES U NDER N ON -C ONVEXITY
We give the following guarantee for pair (A Noisy-GD , ¯A Noisy-GD ) under non-convexity of loss (cid:96) ( θ ; x ) .
Theorem 7 (Accuracy, privacy, deletion, and computation tradeoffs) .
Let constants λ, β, L, σ 2 , η > 0 , q, B > 1 , and 0 < ε dd ≤ ε dp < d .
Let the loss function (cid:96) ( θ ; x ) be σ 2 log( B )-bounded, L -Lipschitz and β -smooth, the regularizer be r ( θ ) =
λ 2 (cid:107) θ (cid:107) 22 , and the weight initialization distribution be ρ = N (cid:16) 0 , σ 2 λ I d (cid:17) .
Then,
both A Noisy-GD and ¯A Noisy-GD are ( q, ε dp ) -RDP for any η ≥ 0 and any K A , K ¯A ≥
if
[Equation]
[Table]
[Equation]
(3.) and all models in sequence (ˆΘ i ) i ≥ 0 output by (A Noisy-GD , ¯A Noisy-GD , Q ) on any D 0 ∈ X n , where Q is an r -requester, satisfy err(ˆΘ i ; D i ) =
˜
O
[Equation]
dq ε dp n 2 + 1 n
[Equation]
qε dd ε dp (cid:19) when inequalities in (12) and (11) are equalities.
Proof of Theorem 7 can be found in Appendix G.5.
The Rényi DP result in (1.) is a restatement of Abadi et al.
(2016, Theorem 1) (discussed further in Appendix G.3.
We prove the deletion and utility results in (2.) and (3.) by building on recent rapid convergence results for Noisy-GD by Vempala & Wibisono (2019); Chewi et al. Speciﬁcally, we show bounds on Rényi divergence of generated models ˆΘ i w.r.t. Gibbs distribution π ( D i ) ∝ exp( −L D i /σ 2 ) for all i ≥ 0 .
Under non-convexity, all prior works on deletion have focused on empirical analysis for utility.
As far as we know, we are the ﬁrst to provide utility guarantees in this setting.
Moreover, our non-convex utility bound exceeds the optimal privacy-preserving utility under convexity by only a factor of ˜ O (cid:0) 1 n (cid:113)
qε dd ε dp (cid:1) , which becomes small for large databases or small deletion to privacy budget ratio.
Also, observe a strict computational beneﬁt in using ¯A Noisy-GD whenever the fraction of edited records in a single update request satisﬁes rn ≤ 12 − ε dd log B .
In the deletion regime where ε dd = log( B ) / 4 , relying on ¯A Noisy-GD rather than retraining with A Noisy-GD is Ω( dn log nr ) cheaper. Remark 4.
The assumption that (cid:96) ( θ ; x ) is L -Lipschitz in both Theorems 6 and 7 can be removed if gradients ∇ (cid:96) (Θ ηk ; x ) computed in step 2 of Algorithm 1 are clipped to L (discussed in Appendix C).
C ONCLUSIONS
We showed that unlearning guarantees are an unsuitable data deletion measure in the online setting of adaptive edit requests and proposed a proper notion of data deletion in line with the "right to be forgotten.
" We also showed that differential privacy is necessary for data deletion when requests are adaptive and proved a reduction from secure deletion under non-adaptive requests to adaptive requests under DP.
Our theoretical results on Noisy-GD based data-deletion, for both convex and non-convex losses, show a substantial computation saving over retraining with a reliable deletion guarantee and at no additional cost in utility.
Appendix Table of Contents
A Table of Notations
B Calculus Refresher
C Loss Function Properties
D Divergence measures and their properties
E Proofs for Section
E.1 Unsoundness and Incompleteness of Ofﬂine Unlearning Deﬁnitions . .
F Proofs for Section {+F.
+}
Contrasting Our Reduction Theorem 5 with Gupta et al.
(2021)’s Results . . . .
G Additional Preliminaries and Proofs for Section
G.1 Langevin Diffusion and Markov Semigroups . .
G.2 Isoperimetric Inequalities and Their Properties . . . . . . . . . . . . . . . . . .
G.3 (Rényi) Differential Privacy Guarantees on Noisy-GD . . . . . . . . . . . . . . G.4 Proofs for Subsection 5.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.5 Proofs for Subsection 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A T ABLE OF N OTATIONS
[Table]
Markov semigroup, its inﬁntesimal generator
[Table]
and its Fokker-Planck operator.
[Table]
Entropy of function f 2 under any arbitrary distribution π .
[Table]
Differential entropy of a distribution.
[Table]
Log-sobolev inequality with constant c .
[Table]
Proxmial mapping for objective L .
B C ALCULUS R EFRESHER
Given a twice continuously differentiable function L : O → R , where O is a closed subset of R d , its gradient ∇L : O → R d is the vector of partial derivatives
[Equation]
Its Hessian ∇ 2 L : O → R d × d is the matrix of second partial derivatives
[Equation]
Its Laplacian ∆ L : O → R is the trace of its Hessian ∇ 2 L , i.e.,
[Equation]
Given a differentiable vector ﬁeld v = ( v 1 , · · · , v d ) : O → R d , its divergence div ( v ) : O → R is
[Equation]
Some identities that we would rely on:
Divergence of gradient is the Laplacian, i
e.,
[Equation]
For any function f : O → R and a vector ﬁeld v : O → R d with sufﬁciently fast decay at the border of O ,
[Equation]
For any two functions f, g : O → R , out of which atleast for one the gradient decays sufﬁcinetly
fast at the border of O , the following also holds.
[Equation]
Based on Young’s inequality, for two vector ﬁelds v 1 , v 2 : O → R d , and any a, b ∈ R such that ab = 1 , the following inequality holds.
[Equation]
Wherever it is clear, we would drop ( θ ) for brevity.
For example, we would represent div ( v ) ( θ ) as only div ( v ) .
C L OSS F UNCTION P ROPERTIES
In this section, we provide the formal deﬁnition of various properties that we assume in the paper.
Let (cid:96) ( θ ; x ) : R d × X → R be a loss function on R d for any record x ∈ X .
Deﬁnition C.
(Lipschitzness) .
A function (cid:96) ( θ ; x ) is said to be L Lipschitz continuous if for all θ, θ (cid:48) ∈ R d and any x ∈ X , |
[Equation]
If (cid:96) ( θ ; x ) is differentiable, then it is L -Lipschitz if and only if ∇ (cid:96) ( θ ; x ) ≤ L for all θ ∈ R d .
Deﬁnition C.2 (Boundedness) .
A function (cid:96) ( θ ; x ) is said to be B -bounded if for all x ∈ X , its output takes values in range [ − B, B ] .
Deﬁnition C.3 (Convexity) .
A continuous differential function (cid:96) ( θ ; x ) is said to be convex if for all θ, θ (cid:48) ∈ R d and x ∈ X ,
[Equation]
and is said to be λ -strongly convex if
[Equation]
Theorem 8 ((Nesterov, 2003, Theorem 2.1.4)) .
A twice continuously differentiable function (cid:96) ( θ ; x ) is convex if and only if for all θ ∈ R d and x ∈ X , its hessian matrix ∇ 2 (cid:96) ( θ ; x ) is positive semideﬁnite, i.e., ∇ 2 (cid:96) ( θ ; x ) (cid:60) 0 and is λ -strongly convex if its hessian matrix satisﬁes ∇ 2 (cid:96) ( θ ; x ) (cid:60) λ I d .
Deﬁnition C.4 (Smoothness) .
A continuously differentiable function (cid:96) ( θ ; x ) is said to be β
-Smooth
if for all θ, θ (cid:48) ∈ R d and x ∈ X ,
[Equation]
Theorem 9 ((Nesterov, 2003, Theorem 2.1.6)) .
A twice continuously differentiable convex function (cid:96) ( θ ; x ) is β -smooth if and only if for all θ ∈ R d and x ∈ X ,
[Equation]
C.0. E FFECT OF G RADIENT C LIPPING
First order optimization methods on a continuously differentiable loss function (cid:96) ( θ ; x ) over a database D ∈ X n with gradient clipping Clip L ( v ) = v / max (cid:16) 1 , (cid:107) v
[Equation]
2 L (cid:17) is equivalent to optimizing
[Equation]
where ¯ (cid:96) ( θ ; x ) is a surrogate loss function that satisﬁes ∇ ¯ (cid:96) ( θ ; x )
=
Clip L ( ∇ (cid:96) ( θ ; x )) .
This surrogate loss function inherits convexity, boundedness, and smoothness properties of (cid:96) ( θ ; x ) , as shown below.
Lemma 10 (Gradient clipping retains convexity) .
If (cid:96) ( θ ; x ) is a twice continuously differentiable convex function for every x ∈ R d , then surrogate loss ¯ (cid:96) ( θ ; x ) resulting from gradient clipping is also convex for every x ∈ R d .
Proof.
Note that the clip operation Clip L ( v ) is a closed-form solution of the orthogonal projection onto a closed ball of radius L and centered around origin, i.e.
[Equation]
By properties of orthogonal projections on closed convex sets, for every v , v (cid:48)
[Equation]
(cid:104)
v
[Equation]
− Clip L ( v ) , v − Clip L
[Equation]
v ) (cid:105)
[Equation]
if and only if (cid:107) v (cid:48) (cid:107)
[Equation] [Equation]
Therefore, for any θ ∈ R d , and x ∈ X , we have
[Equation] [Equation] [Equation]
for all unit vectors ˆ v ∈ R d and magnitude h > 0 .
For the directional derivative of vector ﬁeld ∇ ¯ (cid:96) ( θ ; x ) along ˆ v , deﬁned as ∇ ˆ v ∇ ¯ (cid:96) (
[Equation]
; x
[Equation]
, the above two inequalities imply
[Equation]
31) for all ˆ v .
Therefore, when ∇ ¯ (cid:96) ( θ ; x ) (cid:54) = ∇ (cid:96) ( θ ; x ) , we must have ∇ 2 ¯ (cid:96) ( θ ; x ) = 0 .
And, when ∇ (cid:96) ( θ ; x ) = ∇ ¯ (cid:96) ( θ ; x ) , gradients aren’t clipped, which implies the rate of change of (cid:96) ( θ ; x ) along any direction ˆ v is
[Equation]
θ
[Equation]
x ) h , ˆ v (cid:29) =
[Equation]
∇
[Equation]
(cid:96) ( θ ; x )ˆ v if ∃ h > 0 s.t. ∇ ¯ (cid:96) ( θ + h ˆ v ; x ) = ∇ (cid:96) ( θ + h ˆ v ; x ) otherwise ≥ 0 .
Lemma 11 (Gradient clipping retains boundedness) .
If (cid:96) ( θ ; x ) is a continuously differentiable and
B -bounded function for every x ∈ X , then the surrogate loss ¯ (cid:96) ( θ ; x ) resulting from gradient clipping is also B -bounded.
Proof.
Since (cid:96) ( θ ; x ) is continuously differentiable, its B -boundedness implies path integral of ∇ (cid:96) ( θ ; x ) along any curve between θ, θ (cid:48)
∈ R d is less than 2 B .
Since Clip L ( · ) operation clips the gradient magnitude, the path integral of ∇ ¯ (cid:96) ( θ ; x ) is also less than 2 B .
That is, the maximum and minimum values that ¯ (cid:96) ( θ ; x ) takes differ no more than 2 B .
By adjusting the constant of path integral, we can always ensure ¯ (cid:96) ( θ ; x ) takes values in range [ − B, B ] without affecting ﬁrst order optimization algorithms.
Lemma 12 (Gradient clipping retains smoothness) .
If (cid:96) ( θ ; x ) is a continuously differentiable and β -smooth function for every x ∈ R d , then surrogate loss ¯ (cid:96) ( θ ; x ) resulting from gradient clipping is also β -smooth for every x ∈ R d .
Proof.
Note that the gradient clipping operation is equivalent to an orthogonal projection operation into a ball of radius L , i.e. Clip L ( v ) =
arg min v (cid:48) {(cid:107) v (cid:48) − v (cid:107) 2 : v ∈ R d , (cid:107) v (cid:48) (cid:107) 2 ≤ L } .
Since orthogonal projection onto a closed convex set is a 1 -Lipschitz operation, for any θ, θ (cid:48) ∈ R d , (cid:13)(cid:13) ∇ ¯ (cid:96) ( θ ; x ) − ∇ ¯ (cid:96) ( θ (cid:48) ; x ) (cid:13)(cid:13) 2 ≤ (cid:107)∇ (cid:96)
[Equation]
Additionally, the surrogate loss ¯ (cid:96) ( θ ; x ) is twice differentiable almost everywhere if (cid:96) ( θ ; x ) is smooth, which follows from the following Rademacher’s Theorem.
Theorem 13 (Rademacher’s Theorem (Nekvinda & Zají ˇ cek, 1988)) .
If f : R n → R n is Lipschitz continuous, then f is differentiable almost everywhere in R n .
All our results in Section 5 rely on the above four properties on losses and therefore apply with gradient clipping instead of the Lipschitzness assumption.
D D IVERGENCE MEASURES AND THEIR PROPERTIES
Let
Θ , Θ (cid:48) ∈
O be two random variables with probability measures ν , ν (cid:48) respectively.
We abuse the notataions to denote respective probability densities with ν , ν (cid:48) as well.
We say that ν is absolutely continuous with respect to ν (cid:48)
(denoted by ν (cid:28) ν (cid:48) )
if for all measurable sets O ⊂ O , ν ( O ) =
whenever ν (cid:48)
( O ) = 0 .
Deﬁnition D.1 ( ( ε, δ ) -indistinguishability
(Dwork et al., 2014)) .
We say ν and ν (cid:48) are ( ε, δ ) - indistinguishable if for all O ⊂ O ,
[Equation]
and
[Equation]
In this paper, we measure indistinguishability in terms of Rényi divergence.
Deﬁnition D.2 (Rényi divergence (Rényi et al., 1961)) .
Rényi divergence of ν w.r.t.
ν (cid:48) of order q >is deﬁned as
[Equation]
where E q ( ν (cid:107)
[Equation]
(cid:20)(cid:18) ν
[Equation]
q (cid:21)
[Equation]
34) when ν is absolutely continuous w.r.t.
ν (cid:48) (denoted as ν (cid:28) ν (cid:48) ).
If ν (cid:54)(cid:28) ν (cid:48) , we’ll say R q ( ν (cid:107) ν (cid:48) )
= ∞ .
We abuse the notation R q (Θ (cid:107) Θ (cid:48) ) to denote divergence R q ( ν (cid:107) ν (cid:48) ) between the measures of Θ , Θ (cid:48) .
A bound on Rényi divergence implies a one-directional ( ε, δ ) -indistinguishability as described below.
Theorem 14 (Conversion theorem of Rényi divergence (Mironov, 2017, Proposition 3)) .
Let q >and ε > 0 .
If distributions ν , ν (cid:48) satisfy R q ( ν (cid:107) ν (cid:48) ) <
ε 0 , then for any O ⊂ O ,
[Equation]
(35) for ε = ε 0 + log1 /δ
q − 1 and any 0 < δ < 1 .
We use the following properties of Rényi divergence in some of our proofs.
Theorem 15 (Mononicity of Rényi divergence (Mironov, 2017, Proposition 9)) .
For 1 ≤ q 0 < q , and arbitrary probability measures ν and ν (cid:48) over O , R q 0 ( ν (cid:107) ν (cid:48) )
≤ R q ( ν (cid:107) ν (cid:48) ) .
Theorem 16 (Rényi composition (Mironov, 2017, Proposition 1)) .
If A 1 , · · · , A k are randomized algorithms satisfying, respectively, ( q, ε 1 ) -RDP , · · · , ( q, ε k ) -RDP then their composed mechanism deﬁned as (A 1 ( D ) , · · · , A k ( D )) is ( q, ε 1 + · · · + ε k ) -RDP.
Moreover, i th algorithm can be chosen on the basis of the outputs of algorithms
A 1 , · · · , A i − 1 .
Theorem 17 (Weak triangle inequality of Rényi divergence (Mironov, 2017, Proposition 12)) .
For any distribution ρ on O , the Rényi divergence of ν w.r.t.
ν (cid:48) satisﬁes the following weak triangle inequality:
[Equation]
Another popular notion of information divergence is the Kullback-Leibler divergence.
Deﬁnition D.
(Kullback-Leibler divergence (Kullback & Leibler, 1951)) .
Kullback-Leibler (KL) divergence KL ( ν (cid:107) ν (cid:48) ) of ν w.r.t.
ν (cid:48) is deﬁned as
[Equation]
Rényi divergence generalizes Kullback-Leibler divergence (Van Erven & Harremos, 2014) as
[Equation]
Some other divergence notions that we rely on are the following.
Deﬁnition D.4 (Wasserstein distance (Vaserstein, 1969)) .
Wasserstein distance between ν and ν (cid:48) is
[Equation]
[Equation] where Π is any joint distribution on O × O with ν and ν (cid:48) as its marginal distributions.
Deﬁnition D.5 (Relative Fisher information (Otto & Villani, 2000)) .
If ν (cid:28) ν (cid:48) and νν (cid:48) is differentiable, then relative Fisher information of ν with respect to ν (cid:48) is deﬁned as
[Equation]
Deﬁnition D.6 (Relative Rényi information (Vempala & Wibisono, 2019)) .
Let q > 1 .
If ν (cid:28) ν (cid:48) and νν (cid:48) is differentiable, then relative Rényi information of ν with respect to ν (cid:48) is deﬁned as
[Equation]
∇ (cid:18) ν ( θ ) ν (cid:48)
( θ ) (cid:19) q/ 2 (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)  
 =
E θ ∼ ν (cid:48) (cid:34)(cid:18) ν
( θ ) ν (cid:48)
( θ ) (cid:19) q − 2 (cid:13)(cid:13)(cid:13)(cid:13) ∇ (cid:18) ν ( θ ) ν (cid:48)
( θ ) (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)  (cid:35) .
E P ROOFS FOR S ECTION
Theorem 1.
For every ε > 0 , there exists a pair (A , ¯A) of algorithms that satisfy ( ε, 0) -unlearning under a publish function
f pub such that for all non-adaptive 1 -requesters Q , their exists an adversary that can correctly infer the identity of a record deleted at any arbitrary edit step i ≥ 1 by observing only the post-edit releases φ ≥ i =
( φ i , φ i +1 , · · · ) .
Proof.
For a query h : X → { 0 , 1 } , consider the task of learning the count over a database that is being edited online by a non-adaptive 1 -requester Q .
Since Q is non-adaptive by assumption, it is equivalent to the entire edit sequence { u i }
i ≥ 1 being ﬁxed before interaction.
We design an algorithm pair (A , ¯A) for this task with secret model space being O = N 3 and published outcome space being Φ
= R , with the publish function being f pub ( (cid:104) a, b, c (cid:105) )
= a + b/c + Lap (cid:0)
ε (cid:1) (with the convention that b/c = 0 if b = c = 0 ).
At any step i ≥ 0 , our internal model ˆΘ i = (cid:104) cnt i , del i , i (cid:105) encodes the current count of h on database D i , the count of h on records previously deleted by u ≤ i , and the current step index i .
Our learning algorithm initializes the secret model as ˆ Θ
=
A( D 0 )
= (cid:104) (cid:80) x ∈D
h ( x ) , 0 , 0 (cid:105) , and, for an edit request u i = {(cid:104) ind i , y i (cid:105)} , our algorithm ¯A updates the secret model ˆΘ i − 1 → ˆΘ i following the rule ˆΘ i = ¯A
[Equation] D i − 1 , u i , ˆΘ i − 1 ) =
[Equation]
cnt i , del i , i (cid:105) where
[Equation]
Note that ∀ i ≥ 1 , ∆ i def = del i /i ∈ [0 , 1] .
Therefore, from properties of Laplace mechanism (Dwork et al., 2014), it is straightforward to see that for all
i ≥ 1 ,
[Equation]
Hence, ¯A is an ( ε, 0) -unlearning algorithm for A under f pub .
To show that an adversary can still infer the identity of record deleted by update u i = (ind i , • ) , consider a database D (cid:48)
i − 1 that differs from D i − 1 only at index ind
i such that h ( D (cid:48)
i −
[ind i ]) (cid:54) =
h ( D i −
[ind i ]) .
Let random variable sequences φ ≥
i and φ (cid:48)≥ i denote the releases by ¯A in the scenarios that the ( i − 1) th database was D i − 1 and D (cid:48)
i − 1 respectively.
The divergence between these two random variable sequences reﬂect the capacity of any adversary to infer the record deleted by u i .
Since, we have identical databases after u i , i.e. D j − 1 ◦ u j = D (cid:48)
j − 1 ◦ u j for all j ≥ i , note that both φ j and φ (cid:48)
j are independent Laplace distributions with a shift of exactly 1 j units.
Therefore,
[Equation]
Theorem 2.
There exists a pair (A , ¯A) of algorithms that satisfy (0 , 0) -adaptive-unlearning for an identity publish function f pub ( θ ) = θ such that by designing a 1 -adaptive 1 -requester Q , an adversary, even with no access to Q ’s interaction transcript, can infer the identity of a record deleted at any arbitrary edit step i > 3 with probability at-least 1 − (1 / 2) i − 3 from the post-edit release φ i .
Proof.
Let data universe be X , the internal state space O , and publishable outcome space Φ be R .
Consider the task of releasing a sequence of medians using function med : R ∗ → R in the online setting when the initial database D 0 ∈ X n is being modiﬁed by some adaptive requester Q .
Given a database D ∈ X n , our learning algorithm is deﬁned as A( D ) = med( D ) .
For an arbitrary edit request u ∈ U r , our unlearning algorithm is deﬁned as ¯A( D , u, • ) =
med( D ◦ u ) .
Let the publish function
f
pub : O → Φ be an identity function, i.e. f pub ( θ ) = θ .
For any initial database D 0 ∈ X n and an adaptive sequence ( u i ) i ≥ 1 generated by any ∞ -adaptive 1 -requester Q , note that [Equation] for all i ≥
[Equation]
and any [Equation] ∈ O
[Equation]
Therefore, ¯A is a (0 , 0) -adaptive
unlearning algorithm for A under f pub .
Now suppose that n is odd and D 0 consists of unique entries.
W.L.O.G assume that the median record med( D 0 ) is at index ind m and its owner will be deleting it at step i by sending a non-adaptive edit request u i = {(cid:104) ind m , y (cid:105)} such that y (cid:54) =
med( D 0 ) .
We design the following 1 -adaptive 1 -requester
Q that sends edit requests in the ﬁrst i − 1 steps to ensure with high probability that the published outcome at step i remains the deleted record, i.e., med( D i ) =
med( D 0 ) :
[Equation]
where ind j is randomly sampled from [ n ] \ { ind 1 , · · · , ind j − 1 } without replacement.
Note that by the end of interaction, Q replaces at-least i − 2 unique records in D 0 with φ 0 = med( D 0 ) .
If one of those original records was larger than med( D 0 ) and another was smaller than med( D 0 ) , then it is guaranteed that med( D i ) =
med( D 0 ) .
Therefore, P [med( D i ) =
med( D 0 )] is at-least
P (cid:104) ∃ ind l , ind u ∈ { ind 1 , · · · , ind i − 1 } s.t.
D
[ind l ] < D 
 [ind m ] < D
[ind u ] (cid:105) ≥ 1 − 2 × (cid:18) (cid:98) n (cid:99) /i −(cid:19)(cid:30)(cid:18) n i −(cid:19) ≥ 1 − (cid:18) (cid:19) i −.
E. U NSOUNDNESS AND I NCOMPLETENESS OF O FFLINE U NLEARNING D EFINITIONS
In this subsection, we show that our criticisms on both soundness and completeness of unlearning notions in Section 3 also apply to the one-stage unlearning deﬁnitions of Guo et al. ; Sekhari et al. ; Ginart et al.
Deﬁnition E.1 ( ( ε, δ ) -certiﬁed
removal (Guo et al., 2019)) .
A removal mechanism ¯A performs ( ε, δ ) -certiﬁed removal for learning algorithm
A if for all databases D ⊂ X and deletion subset
S ⊂ D , ¯A
[Equation]
Deﬁnition E.2 ( ( ε, δ ) -unlearning (Sekhari et al., 2021)) .
For all D ⊂ X of size n and deletion subset S ⊂ D such that | S | ≤ m , a learning algorithm A and an unlearning algorithm ¯A is ( ε, δ ) -unlearning if
[Equation]
where ∅ denotes the empty set and T ( D ) denotes the data statistics available to ¯A regarding D .
{+Deﬁnition E.
+}
(Data Deletion Operation Ginart et al.
Fix any dataset D ⊂ X and learning algorithm A .
{+Operation ¯A is a deletion operation for A if ¯A( D , S, ¯A( D )) 0 , 
+}
≈ A( D \ S ) for any set
S ⊂ D selected independently of A( D ) .
Unsoundness.
Deﬁnitions E.1 and E.2 make no assumptions about dependence between the deletion request S and the learned model
A( D ) .
So, request S can depend on A( D ) .
This dependence is common in the real world; for example, a user deletes her information if she doesn’t like what model
A( D ) reveals about her.
We present the following construction along the same lines as our proof in
Theorem 2 to show that Deﬁnitions E.1 and E.2, are unsound.
For the universe of records X = {− 2 , − 1 , 1 , 2 } , consider the following learning and unlearning algorithms:
[Equation]
Note that for any D ⊂ X and any S ⊂ D , the above algorithm pair (A , ¯A) satisﬁes Deﬁnitions E.1, E.2 and E.3 for ε = δ = 0 and T ( D )
=
D .
Suppose the adversary is aware that the following dependence holds between the learned model
A( D ) and deletion request S :
[Equation]
Consider two neighbouring databases D − 1 = {− 2 , − 1 , 2 } and D 1 = {− 2 , 1 , 2 } .
Knowing the above dependence, an adversary can determine whether D = D − 1 or D = D 1 by looking only at ¯A( D , S, A( D )) .
This is because if D = D − 1 , then the observation after unlearning is 2 , and if
D
= D 1 , the observation after unlearning is − 2 .
So, even though (A , ¯A) satisﬁes the guarantees of
Guo et al.
(2019) and Sekhari et al.
(2021), it blatantly reveals the identity ( − 1 or 1 ) of a deleted record to an adversary observing only the post-deletion release.
Note that Deﬁnition E.3 assumes that the requests S is selected independently of the learned model
A( D ) .
So, our construction does not apply, keeping the possibility that their deﬁnition is sound.
We remark, however, that algorithms satisfying their deﬁnitions cannot be trusted in settings where we expect some dependence between deletion requests and the learned models.
Incompleteness.
Deﬁnitions E.1, E.2 and E.3 are also incomplete.
Consider an unlearning algorithm ¯A that outputs a ﬁxed output x 1 ∈ X if the deletion request S = ∅ and outputs another ﬁxed output x 2 ∈ X if the deletion request S (cid:54) = ∅ .
It is easy to see that ¯A is a valid deletion algorithm as its output does not depend on the input database D or
the learned model
A( D ) .
However, note that ¯A does not satisfy the unlearning Deﬁnition E.2, for any learning algorithm A .
And, for a learning algorithm A( D ) =
(cid:80) x ∈D x , one can also verify that the pair (A , ¯A) does not satisfy Deﬁnitions E. and E.3 either.
F P ROOFS FOR S ECTION
Theorem 3 (Data-deletion Deﬁnition 4.1 is sound) .
If the algorithm pair (A , ¯A) satisﬁes ( q, ε ) - data-deletion guarantee under all p -adaptive r -requesters, then even with the power of designing an p -adaptive r -requester Q that interacts with the curator before deletion of a target record at any step i ≥ 1 , any adversary observing only the post-deletion releases (ˆΘ i , ˆΘ i +1 , · · · ) has its membership inference advantage for inferring a deleted target bounded as
[Equation]
Proof.
For an arbitrary step i ≥ 1 , suppose one of the replacement operations in the edit request u
i ∈
U r replaces a record at index ‘ ind ’ from the database D
i − 1 with ‘ y ’.
{+In the worst case, this record D i − 1 [ind] might have been there from the start, i.e. D 
+}
{+[ind] = D 
+}
[ind] , and inﬂuenced all the decisions of the adaptive requester Q in the edit steps 1 , · · · , i − 1 .
{+To prove soundness, we need to show that if (A , ¯A) satisﬁes ( q, ε ) -data-deletion, then even in this worst-case scenario, no adaptive adversary can design a membership inference test MI (ˆΘ i , ˆΘ i +1 , · · · ) ∈ { 0 , 1 } that can distinguish with high probability the null hypothesis H 
+}
{+= {D 0 [ind] = x } from the alternate hypothesis H 1 = {D 
+}
[ind] = x (cid:48) } for any x , x (cid:48) ∈ X .
That is, the advantage of any test MI, deﬁned as
[Equation]
must be small.
Since after processing edit request u i , the databases D i , D i +1 , · · · no longer contain the deleted record D i − 1 [ind] , the data-processing inequality implies that future models ˆΘ i +1 , ˆΘ i +2 , · · · cannot have more information about D i − 1 [ind] that what is present in ˆΘ i .
Therefore, any test
MI (ˆΘ i , ˆΘ i +1 , · · · ) has a smaller advantage than the optimal test MI ∗ (ˆΘ i ) ∈ { 0 , 1 } that only uses ˆΘ i .
Also, since (A , ¯A) satisfy ( q, ε ) -data-deletion for any p -adaptive r -requester Q , we know from
Deﬁnition 4.1 that there exists a mapping π Q i such that for all D 0 ∈ X n , the model ˆΘ i generated by the interaction between (A , ¯A , Q ) on D 0 after i th edit satisﬁes R q (cid:16) ˆΘ i (cid:13)(cid:13)(cid:13) π Q i ( D 0 ◦ (cid:104) ind , y (cid:105) ) (cid:17) ≤ ε .
As the database D 0 ◦(cid:104) ind , y (cid:105) is identical under both hypothesis H 0 and H 1 , we have R q (cid:16) ˆΘ i | H b (cid:13)(cid:13) (cid:13) ¯Θ (cid:17) ≤ ε for b ∈ { 0 , 1 } ,
where ¯Θ ∼ π Q i ( D 0 ◦ (cid:104) ind , y (cid:105) ) .
From Rényi divergence to ( ε, δ ) -indistinguishability conversion described in Remark 1, we get
[Equation]
where ε (cid:48) ( δ ) =
ε + log1 /δ
q − 1 for any 0 < δ < 1 .
On adding the two inequalities, we get:
[Equation]
Theorem 4 (DP is necessary for data-deletion) .
If learning algorithm A : X n → O is not (0 , δ ) -DP with respect to the replacement of a single record and deletion
algorithm ¯A :
X n × U × O
→ O is not (0 , δ ) -DP with respect to the replacement of a single record that is not being deleted, then the pair (A , ¯A) cannot satisfy ( q, δ 4 / 2) -data-deletion under 1 -adaptive 1 -requester for any q > 1 .
Proof.
If A is not (0 , δ ) -DP with respect to replacement of a single record, then there exists a pair of neighbouring databases D , D (cid:48) such that
[Equation]
{+Similarly, if ¯A is not (0 , δ ) -DP with respect to replacement of a single record that is not being deleted, then there exists a pair of databases ¯ D , ¯ D (cid:48) and an edit request ¯ u ∈ U 
+}
such that ¯ D ◦ ¯ u and ¯ D (cid:48) ◦ ¯ u are neighbouring and for all θ ∈ O ,
[Equation]
Since TV distance is bounded from below in both cases, there exists tests Test , Test : O → { 0 , 1 } such that
[Equation]
and for all θ ∈ O ,
[Equation]
Assume W.L.O.G. that ¯ u replaces at index n and the edited databases ¯ D ◦ u, ¯ D (cid:48) ◦
u still differs at index 1 .
Also assume that D , D (cid:48) differs at index n .
Recall from Deﬁnition 4.1 that satisfying ( q, δ 4 2 ) -data-deletion under 1 -adaptive 1 -requesters requires existence of a map π Q n : X n → O for each Q such that for all D 0 ∈ X n ,
[Equation]
To prove the theorem statement, we show that for a starting database D 0 ∈ {D , D (cid:48) } and an edit request u n = ¯ u that deletes the differing record in choices of D 0 at edit step n , there exists a 1 -adaptive 1 -requester Q that sends adaptive edit requests u 1 , · · · , u n − 1 in the ﬁrst n − 1 steps such that no map π Q n exists that satisﬁes (55) for both choices of D 0 .
{+Consider the following construction of 1 -adaptive 1 -requester Q that only observes the ﬁrst model ˆΘ 
+}
=
A( D 0 ) and generates the edit requests ( u 1 , · · · , u n − 1 ) as follows:
[Equation]
] (cid:105) otherwise .
This requester Q transforms any initial database D 0 to D n − 1 = ¯ D if the outcome Test (ˆΘ 0 )
= 1 , otherwise to D n − 1 = ¯ D (cid:48) .
Consider an adversary that does not observe the interaction transcript (ˆΘ <n ; u <n ) , but is interested in identifying whether D 0 was D or D (cid:48) .
The adversary gets to observe only the output ˆΘ n = ¯A( D n − 1 , u n , ˆΘ n − 1 ) generated after processing the edit request u n = ¯ u .
On this observation
[Equation]
the adversary runs the membership inference test MI (ˆΘ n ) =
Test (ˆΘ n ) .
The membership inference advantage of MI is
[Equation]
{+So, it must be true that the total variation distance between ˆΘ n given D 
+}
{+= D and D 
+}
= D (cid:48) is lower bounded as
[Equation]
[Equation]
From triangle inequality of total variation distance, Pinsker’s inequality
[Equation]
and monotonicity of Rényi divergence w.r
[Equation]
t. order q , note that for a random variable ¯Θ ∈ O with any arbitrary distribution,
[Equation]
This implies that for all random variables ¯Θ , and all q > 1 ,
[Equation]
But, to satisfy ( q, δ 4 2 ) -data-deletion under 1 -adaptive 1 -requesters, there must exist a mapping π Q n for which (55) must hold for both choices of D 0 ∈ {D , D (cid:48) } .
Since the random variable π Q n ( D 0 ◦ u n ) is identically distributed in our construction for both choices of D 0 ∈ {D , D (cid:48) } , from (58) we have that no such map π Q n can exist.
Theorem 5 (Reduction from Adaptive to Non-adaptive Data Deletion) .
If an algorithm pair (A , ¯A) satisfy ( q, ε dd ) -data-deletion under all non-adaptive r -requesters and ( q, ε dp ) -Rényi differential privacy, then it also satisﬁes ( q, ε dd + pε dp ) -data-deletion for all p -adaptive r -requesters.
Proof.
To prove this theorem, we need to show that for any p -adaptive r -requester Q , there exists a construction for a map π Q i : X n → O such that for all D 0 ∈ X n , the sequence of model (ˆΘ i ) i ≥generated by the interaction between ( Q , A , ¯A) on D 0 satisﬁes the following inequaltiy for all i ≥ 1 : [Equation]
[Equation] Fix a database D 0 ∈ X n and an edit request u i ∈
U r . Let D (cid:48)
∈ X n be a neighbouring database deﬁned to be D {+(cid:48) 
+}
= D 0 ◦ (cid:104) ind , y (cid:105) for an arbitrary replacement operation (cid:104) ind , y (cid:105) ∈ u i .
Given any p -adaptive r -requester Q , let (ˆΘ i ) i ≥
and ( U i )
i ≥ 1 be the sequence of released model and edit request random variables generated on Q ’s interaction with (A , ¯ A) with inital database as D 0 .
Similarly, let (ˆΘ (cid:48) i ) i ≥ 0 and ( U (cid:48) i ) i ≥ 1 be the corresponding sequences generated due to the interaction among ( Q , A , ¯A) on D (cid:48) 0 .
Since (A , ¯A) is assumed to satisfy ( q, ε dd ) -data-deletion guarantee under non-adpative r -requesters, recall from Remark 3 that there exists a mapping π : X n → O such that for any ﬁxed edit sequence u ≤ i def = ( u 1 , u 2 , · · · , u i ) ,
[Equation]
Note that since the replacement operation (cid:104) ind , y (cid:105) is part of the edit request u i , we have D 0 ◦ U (cid:48) <
i ◦ u i = D (cid:48) 0 ◦ U (cid:48) <
i ◦ u i .
Moreover, since the sequence U (cid:48) <i of edit requests is generated by the interaction of ( Q , A , ¯A) on D (cid:48) {+
+}
= D 0 ◦ (cid:104) ind , u (cid:105) and the i th edit request u i is ﬁxed beforehand, we can deﬁne a valid construction of[-D 
 [ind u ] as the random variable sub-sequence U (cid:48) i< 0 was generated on-] a [-dataset D (cid:48) 
-]map π Q i : X n → O as per Deﬁnition 4.1 as follows:
[Equation]
For brevitly, let ˆΘ u =
¯A
( D 0 ◦
U <
i , u i , ˆΘ i − 1 )
[Equation]
and ˆΘ (cid:48)
u
= π Q i ( D 0 ◦ (cid:104) ind , y (cid:105) ) .
For this construction, we prove the requisite bound in (59) as follows.
[Equation]
[Equation]
[Equation]
are joint PDFs of (ˆΘ u , U <i ) & (ˆΘ
[Equation]
, U
[Equation]
U (cid:48)
[Equation]
i ) ≤
[Equation]
(cid:17)(cid:13)(cid:13)(cid:13)(cid:16) ˆΘ
[Table] [Equation] [Table]
[Equation]
[Table] p (cid:17)(cid:17) (If Q sees outputs at steps s 1 , · · · , s p ) ≤ ε dd + pε dp .
(Via Rényi composition)
F. C ONTRASTING O UR R EDUCTION T HEOREM 5 WITH G UPTA ET AL . (2021)’ S R ESULTS
The recent work of Gupta et al.
(2021), also studies adaptive data deletion and proves a reduction from adaptive unlearning guarantee to non-adaptive unlearning guarantee in Deﬁnition 2.6 under differential privacy.
We remark that the reduction Theorem 3.1 by Gupta et al.
(2021) relies on DP with regards to a change in the description of learning/unlearning algorithm’s coins and not with regards to the standard replacement of records.
In contrast, our Theorem 5 presents a reduction from adaptive to non-adaptive data-deletion guarantee under DP with respect to the standard replacement of records.
We emphasize that these two Theorems are fundamentally different.
The adaptive unlearning deﬁnition of Gupta et al.
(2021) is designed to ensure with a high probability that no adaptive requester Q can force the output distribution of the unlearning algorithm ¯A
( D i − 1 , u i , ˆΘ i − 1 ) to diverge substantially from that of retraining
algorithm A( D i ) .
Such an attack is possible in stateful unlearning algorithms that rely on persistent structures that are only randomized once during initialization, for example, the initial partitioning of start database D 0 in Bourtoule et al.
(2021)’s SISA algorithm.
Gupta et al.
(2021) show in their Theorem 5.1 that an adaptive update requester Q can interactively send deletion requests u 1 , · · · , u i to SISA so that the partitioning of remaining records in D i
= D 0 ◦ u 1 · · · u i follows a pattern that is unlikely to occur on repartitioning of D i when executing A( D i ) .
As proved in their reduction in Theorem 3.1, a straightforward way to prevent this is by ensuring that the uncertainty regarding the persistent structures remains private for long periods of time to an adversary observing the unlearned model.
Hence the proof of their reduction from adaptive unlearning guarantee to non-adaptive unlearning guarantee relies on DP with regards to the coins of the unlearning algorithm.
Our work shows that satisfying adaptive unlearning deﬁnition of Gupta et al.
(2021) still does not guarantee data deletion.
In Theorem 2, we demonstrate that there exists an algorithm pair (A , ¯A) satisfying (a strictly stronger version) of adaptive unlearning (Gupta et al., 2021, Deﬁnition 2.3), but still causes blatant non-privacy of deleted records in post-deletion release.
The vulnerability we identify occurs because an adaptive requester can learn the identity of any target record before it is deleted and re-encode it back in the curator’s database by sending edit requests.
Because of this, an adversary (who knows how the adaptive requester works but does not have access to the requester’s interaction transcript) can extract the identity of the target record from the model released after processing the deletion request.
In our work, we argue that a reliable (and necessary) way to prevent this attack is to make sure that no adaptive requester ever learns the identity of a target record from the p pre-deletion model releases it has access to.
Consequently, our reduction in Theoremfrom adaptive to non-adaptive requests relies on differential privacy with respect to the replacement of records instead.
G A DDITIONAL P RELIMINARIES AND P ROOFS FOR S ECTIONG. L ANGEVIN D IFFUSION AND M ARKOV S EMIGROUPS
Langevin diffusion process on R d with noise variance σ 2 under the inﬂuence of a potential
L : R d → R is characterized by the Stochastic Differential Equation (SDE)
[Equation]
(63) where d Z t =
Z t +d t − Z t ∼ √ d t N (0 , I d ) is the d -dimensional Weiner process.
We present some preliminary knowledge on the diffusion theory used in our analysis.
Let p t ( θ 0 , θ t ) denote the probability density function describing the distribution of Θ t , on starting from Θ
= θ 0 at time t = 0 .
For SDE (63), the associated Markov Semigroup P , is deﬁned as a family of operators ( P t ) t ≥ 0 , such that an operator P t sends any real-valued bounded measurable function f : R d → R to
[Equation]
The infentisimal generator G def = lim s → 0 1 s [ P t + s − P s ] for this diffusion Semigroup is
G f = σ 2 ∆ f − (cid:104)∇L , ∇ f (cid:105) .
This generator G , when applied on a function f ( θ t ) , gives the infentisimal change in the value of a function f when θ t undergoes diffusion as per (63) for d t time.
That is,
[Equation]
[Equation] The dual operator of G is the Fokker-Planck operator G ∗ , which is deﬁned as the adjoint of generator
G , in the sense that
[Equation]
for all real-valued bounded measurable functions f, g : R d → R .
Note from (66) that, this operator provides an alternative way to represent the rate of change of function f at time t : [Equation]
[Equation]
To put it simply, Fokker-Planck operator gives the infentesimal change in the distribution of Θ t with respect to time.
For the Langevin diffusion SDE (63), the Fokker-Planck operator is the following:
[Equation]
From this Fokker-Placnk equation, one can verify that the stationary or invariant distribution π of
Langevin diffusion, which is the solution of ∂ t p t = 0 , follows the Gibbs distribution
[Equation]
Since π is the stationary distribution, note that for any measurable bounded function f : R d → R ,
[Equation]
G. I SOPERIMETRIC I NEQUALITIES AND T HEIR P ROPERTIES
Convergence properties of various diffusion semigroups have been extensively analyzed in literature under certain isoperimetric assumptions on the stationary distribution π (Bakry et al., 2014).
One such property of interest is the logarithmic Sobolev ( LS ) inequality (Gross, 1975), which we deﬁne next.
The carré du champ operator Γ of a diffusion semigroup with invariant measure µ is deﬁned using its inﬁntesimal generator G as
[Equation]
72) for every f, g ∈ L 2 ( µ ) .
Carre du champ operator represent fundamental properties of a Markov semigroup that affect its convergence behaviour.
One can verify that Langevin diffusion semigroup’s carre du champ operator (on differentiable f, g ) is
[Equation]
We use shorthand notation Γ( f ) =
Γ( f, f [Equation]
= σ 2 (cid:107)∇ f (cid:107) [Equation]
Deﬁnition G.1 (Logarithmic Sobolev Inequality (see Bakry et al.
(2014, p. 24))) .
A distribution with probability density π is said to satisfy a logarithmic Sobolev inequality ( LS( c ) ) (with respect to Γ in (73) ) if for all functions f ∈ L 2 ( µ ) with continuous derivatives ∇ f ,
[Equation]
(74) where entropy Ent π is deﬁned as
[Equation]
Logarithmic Sobolev inequality is a very non-restrictive assumption and is satisﬁed by a large class of distributions.
The following well-known result show that Gaussians satisfy LS inequality.
Lemma 18 (LS inequality of Gaussian distributions (see Bakry et al.
(2014, p. 258))) .
Let ρ be a
Gaussian distribution on R d with covariance σ 2 /λ (i.e., the Gibbs distribution (70) with L ( · ) being the L 2 regularizer r ( θ ) =
λ 2 (cid:107) θ (cid:107) 22 ).
Then ρ satisﬁes LS( λ ) tightly (with respect to Γ in (73) ), i.e.
[Equation]
Additionally, if µ is a distribution on R d that satisfy LS( c ) , then the convolution µ (cid:126) ρ , deﬁned as the distribution of Θ + Z where Θ ∼ µ and Z ∼ π , satisﬁes LS inequality with constant (cid:0) 1 c + 1 λ (cid:1) − 1 .
Bobkov (2007) show that like Gaussians, all strongly log concave distributions (or more generally, logconcave distributions with ﬁnite second order moments) satisfy LS inequality (e.g. Gibbs distribution π with any strongly convex L ).
LS inequality is also satisﬁed under non-log-concavity too.
For example, LS inequality is stable under Lipschitz maps, although such maps can destroy log-concavity.
Lemma 19 (LS inequality under Lipschitz maps (see Ledoux (2001))) .
If π is a distribution on R d that satisﬁes LS( c ) , then for any L -lipschitz map T : R d → R d , the pushforward distribution T # π , representing the distribution of T (Θ) when Θ ∼ π , satisﬁes LS( c/L 2 ) .
LS inequality is also stable under bounded perturbations to the distribution, as shown in the following lemma by Holley & Stroock (1986).
Lemma 20 ( LS inequality under bounded perturbations (see Holley & Stroock (1986))) .
If π is the probability density of a distribution that satisﬁes LS( c ) , then any proability distribution with density π (cid:48) such that 1 √ B ≤ π ( θ ) π (cid:48) ( θ ) ≤ √ B everywhere in R d for some constant B > 1 satisﬁes LS( c/B ) .
Logarithmic Sobolev inequality is of interest to us due to its equivalence to the following inequalities on Kullback-Leibler and Rényi divergence.
Lemma 21 ( LS inequality in terms of KL divergence (Vempala & Wibisono, 2019)) .
The distribution π satisﬁes LS( c ) inequality (with respect to Γ in (73) )
if and only if for all distributions µ on R d such that µπ ∈ L 2 ( π ) with continuous derivatives ∇ µπ ,
[Equation]
Proof.
Set f 2 in (74) to µπ to obtain (77).
Alternatively, set µ = f 2 π E π
[ f 2 ] in (77) to obtain (74).
Lemma 22 (Wasserstein distance bound under LS inequality (Otto & Villani, 2000, Theorem 1)) .
If distribution π satisﬁes LS( c ) inequality (with respect to Γ in (73) )
then for all distributions µ on R d ,
[Equation]
Lemma 23 ( LS inequality in terms of Rényi Divergence (Vempala & Wibisono, 2019)) .
The distribution π satisﬁes LS( c ) inequality (with respect to Γ in (73) )
if and only if for all distributions µ on R d such that µπ ∈ L 2 ( π ) with continuous derivatives ∇ µπ , and any q > 1 ,
[Equation]
Proof.
For brevity, let the functions R ( q )
=
R q ( µ (cid:107) π ) , E ( q ) =
E q ( µ (cid:107) π ) , and I ( q ) =
I q ( µ (cid:107) π ) .
Let function f 2 ( θ ) =
(cid:16) µ ( θ ) π ( θ ) (cid:17) q .
Then,
[Equation]
[Equation] ) and,
[Equation]
(From Lebniz rule and [Equation]
Moreover,
[Equation]
On substituting (74) with the above equalities
[Equation]
we get:
[Equation]
G. (R ÉNYI ) D IFFERENTIAL P RIVACY G UARANTEES ON N OISY -GD
In this section, we recap the differential privacy bounds in literature for Noisy-GD Algorithm 1.
Theorem 24 (Rényi DP guarantee for Noisy-GD Algorithm 1) .
If (cid:96) ( θ ; x ) is L -lipschitz, then NoisyGD satisﬁes ( q, ε ) -RDP with ε = qL 2 σ 2 n 2 · ηK .
Proof.
The L 2 sensitivity of gradient ∇L D ( θ ) def =
[Equation]
n
[Equation]
∈D ∇ (cid:96) (
[Equation]
; x ) + ∇ r ( θ ) computed in stepof Algorithm 1 for neighboring databases in X n that differ in a single record is 2 Ln since (cid:96) ( θ ; x ) is
L -Lipschitz.
Conditioned on observing the intermediate model Θ ηk
= θ k at step k , the next model Θ η ( k +1) after the noisy gradient update is a Gaussian mechanism with noise variance 2 σ 2 /η .
So, for neighboring databases D , D (cid:48) ∈
X n , we have from the Rényi DP bound of Gaussian mechanisms proposed by Mironov (2017, Proposition 7) that
[Equation]
where (Θ ηk ) 0 ≤ k ≤ K and (Θ (cid:48) ηk ) 0 ≤ k ≤ K are intermediate parameters in Algorithm 1 when run on databases D and D (cid:48) respectively.
Finally, from Rényi composition Mironov (2017, Proposition 1), we have
[Equation]
Remark 5.
Different papers discussing Noisy-GD variants adopt different notational conventions for the total noise added to the gradients.
The noise variance in our Algorithm 1 is 2 ησ 2 ; but is η 2 σ 2 L 2 nin the full-batch setting of DP-SGD by Abadi et al.
To translate the bound in
Theorem 24, one can simply rescale σ across different conventions to have the same noise variance, i.e., 2 ησ 2 = η 2 ˆ σ 2 L 2 n.
Our Theorem 24 is somewhat identical to Abadi et al.
(2016)’s ( ε, δ )
-DP bound.
{+To verify this, note from Rényi divergece to ( ε, δ ) -indistinguishability conversion discussed in Remark 
+}
that (1 + 2 ε log 1 δ , ε 2 ) -Rényi
DP implies ( ε, δ ) -DP.
So, setting the bound in Theorem 24 to be smaller than ε 2 and substituting q = 1 + 2 ε log 1 δ , we get
[Equation]
For ε ≤ 2 log 1 δ , we get the same noise bound as in Abadi et al.
(2016, Theorem 1) for their (full-batch)
DP-SGD algorithm.
Next, we recap the tighter Rényi DP guarantee of Chourasia et al.
(2021) under stronger assumptions on the loss function.
Theorem 25 (Rényi DP guarantee for Noisy-GD Algorithm 1 (Chourasia et al., 2021)) .
{+If (cid:96) ( θ ; x ) is convex, L -Lipschitz, and β -smooth and r ( θ ) is the L 2 regularizer with constant λ , then Noisy-GD with learning rate η < 1 β + λ satisﬁes ( q, ε ) -RDP with ε = 4 qL 2 λσ 2 n 2 (cid:0) 
+}
− e − ληK/ 2 (cid:1) .
G. P ROOFS FOR S UBSECTION 5.
In this appendix, we provide a proof of our Theorem 6 which applies to convex losses (cid:96) ( θ ; x ) under
L 2 regularizer r ( θ ) .
Let D 0 ∈ X n be any arbitrary database, and Q be any non-adaptive r -requester.
Our ﬁrst goal in this section is to prove ( q, ε dd ) -data-deletion guarantees on our proposed algorithm pair (A Noisy-GD , ¯A Noisy-GD ) (in Deﬁnition 5.1) under Q .
That is, if (ˆΘ i )
i ≥ 0 is the sequence of models produced by the interaction between (A Noisy-GD , ¯A Noisy-GD , Q ) on D 0 , we need to show that their exists a mapping π Q i such that for all i ≥ 1 and any u i ∈ U r ,
[Equation]
for all (cid:104) ind , y (cid:105) ∈ u i .
[Equation]
For an arbitrary replacement operation (cid:104) ind , y (cid:105) in u i , we deﬁne a map π Q i ( D 0 ◦(cid:104) ind , y (cid:105) )
[-i − 1 existed in the start database D 
 (i.e. ind u (cid:54)∈ u j for all 1 ≤ j ≤ i − 1 ). For a neighbouring database D (cid:48) 0 ∈ X n , such that D 0 ◦ u-] = ˆΘ (cid:48)
i , where the model sequence (ˆΘ (cid:48) i ) i ≥ 0 is produced by the interaction of between (A Noisy-GD , ¯A Noisy-GD , Q ) on
D 0 ◦ (cid:104) ind , y (cid:105) .
Since non-adaptive requester Q is equivalent to ﬁxing the edit sequence ( u i ) i ≥a-priori, note that showing the data-deletion guarantee reduces to proving the following DP-like bound
[Equation]
for for all u ≤ i and for all neighbouring databases D 0 , D (cid:48)[-
-]
[Equation] s [Equation] t
[Equation] D
[Equation]
= D [Equation] ◦ (cid:104) ind , y (cid:105) with (cid:104) ind , y (cid:105) ∈ u i .
Note from our Deﬁnition 5.1 that the sequence of models (ˆΘ 0 , · · · , ˆΘ i ) can be seen as being generated from a continuous run of Noisy-GD, where:
1. for iterations 0 ≤ k < K A , the loss function is L D 0 , 2.
for the iterations K A + ( j − 1) K ¯A ≤ k < K A + jK ¯A on any 1 ≤ j ≤ i − 1 , the loss function is L D j , and 3.
for the iterations K A + ( i − 1) K ¯A ≤ k < K A + iK ¯A , the loss function is L D i − 1 ◦ u i .
Let (Θ ηk ) 0 ≤ k ≤ K A + iK ¯A be the sequence representing the intermediate parameters of this extended Noisy-GD run.
Similarly, let (Θ (cid:48) ηk ) k ≥ 0 be the parameter sequence corresponding to the extended run on the neighbouring database D (cid:48) 0 . Since (cid:104) ind , y (cid:105) ∈ u i , note from the construction that D (cid:48) i − 1 ◦ u i =
D i − 1 ◦ u i , meaning that the loss functions while processing request u i is identical for the two processes, i.e. L D i − 1 ◦ u i =
L D (cid:48) i − 1 ◦ u i . For brevity, we refer to the database seen in iteration k of the two respective extended runs as D ( k ) and D (cid:48) ( k ) respectively.
In short, these two discrete processes induced by Noisy-GD follow the following update rule for any 0 ≤ k < K A + iK ¯A :
[Equation]
where Z k , Z (cid:48)
k ∼ N (0 , I d ) , (83) and Θ 0 and Θ (cid:48) 0 are sampled from same the weight initialization distribution ρ .
To prove the bound in (82), we follow the approach proposed in Chourasia et al.
(2021) of interpolating the two discrete stochastic process of Noisy-GD with two piecewise-continuous tracing diffusions Θ t and Θ (cid:48) t in the duration ηk < t ≤ η ( k + 1) , deﬁned as follows. [Equation]
[Equation]
where Z t , Z (cid:48) t are two independent Weiner processes, and T k is a map on R d deﬁned as
[Equation]
[Equation] Note that equation (84) is identical to (83) when t = η ( k + 1) , and can be expressed by the following stochastic differential equations (SDEs):
[Equation]
where g k (Θ) = 1 2 n ∇ [ (cid:96) (Θ; D ( k ) [ind]) − (cid:96) (Θ; D (cid:48)
( k )[ind])] , (86) and initial condition lim t → ηk + Θ t =
T k (Θ ηk ) , lim t → ηk + Θ (cid:48) t =
T k (Θ (cid:48) ηk ) .
These two SDEs can be equivalently described by the following pair of Fokker-Planck equations.
Lemma 26 (Fokker-Planck equation for SDE (86)) .
Fokker-Planck equation for SDE in (86) at time ηk < t ≤ η ( k + 1) , is
[Equation]
where µ t and µ (cid:48) t are the densities of Θ t and Θ (cid:48) t respectively. [Figure]
Proof.
Conditioned on observing parameter Θ ηk
= θ ηk , the process (Θ t ) ηk<t ≤ η ( k +1) is a Langevin diffusion along a constant Vector ﬁeld (i
[Figure]
e. on conditioning, we get a Langevin SDE (63) with ∇L ( θ ) =
g k
( θ ηk ) for all θ ∈ R d ).
Therefore as per (69), the conditional probability density µ t | ηk ( ·| θ ηk ) of Θ t given Θ ηk follows the following Fokker-Planck equation:
[Equation]
Taking expectation over µ ηk which is the distribution of Θ ηk ,
[Equation]
where µ ηk, | t is the conditional density of Θ ηk given Θ t .
Proof for second fokker-planck equation is similar.
We provide an overview of how we bound equation (82) in Figure 1.
Basically, our analysis has two phases; in phase (I) we provide a bound on R q (cid:16) ˆΘ i − 1 (cid:13)(cid:13)(cid:13) ˆΘ (cid:48)
i − 1 (cid:17) that holds for any choice of number of iterations K A and K ¯A , and in phase (II) we prove an exponential contraction in the divergence
R q (cid:16) ¯A( D i − 1 , u i , ˆΘ i − 1 (cid:13)(cid:13)(cid:13) ¯A
( D (cid:48)
i − 1 , u i , ˆΘ (cid:48)
i − 1 ) (cid:17) with number of iterations K ¯A .
We ﬁrst introduce a few lemmas that will be used in both phases.
The ﬁrst set of following lemmas show that the transformation Θ ηk , Θ (cid:48)
ηk → T k (Θ ηk ) , T k
(Θ ηk ) preserves the Rényi divergence.
To prove this property, we show that T k is a differentiable bijective map in Lemma 28 and apply the following Lemma from Vempala & Wibisono (2019).
Lemma 27 (Vempala & Wibisono (2019, Lemma 15)) .
If T : R d → R d is a differentiable bijective map, then for any random variables Θ , Θ (cid:48) ∈ R d , and for all q > 0 ,
[Equation]
Lemma 28.
If (cid:96) ( θ ; x ) is a twice continuously differentiable, convex, and β -smooth loss function and regularizer is r ( θ )
= λ 2 (cid:107) θ (cid:107) 22 , then the map T k deﬁned in (85) is:
a differentiable bijection for any η < 1 λ + β , and 2.
(1 − ηλ ) -Lipschitz for any η ≤ 2 2 λ + β .
Proof.
Differentiable bijection.
To see that T k is injective, assume T k ( θ )
=
T k ( θ (cid:48) ) for some θ, θ (cid:48) ∈ R d .
Then, by ( β + λ ) -smoothness of L def =
( L D ( k ) + L D (cid:48) ( k
[Equation]
[Equation]
Since η < 1 / ( λ + β ) , we must have (cid:107) θ − θ (cid:48) (cid:107) 2 = 0 .
For showing T k is surjective, consider the proximal mapping
[Equation]
Note that prox L ( · ) is strongly convex for η < 1 λ + β .
Therefore, from KKT conditions, we have
[Equation]
Differentiability of T k follows from the twice continuously differentiable assumption on (cid:96) ( θ ; x ) .
Lipschitzness.
Let L def =
( L D ( k ) + L D (cid:48) ( k ) ) / 2 .
For any θ, θ (cid:48) ∈ R d ,
[Equation]
We deﬁne a function g ( θ ) =
L ( θ ) − λ 2 (cid:107) θ (cid:107) 22 , which is convex and β -smooth.
By co-coercivity property of convex and β -smooth functions, we have
[Equation]
Substituting this in the above inequality, and noting that η ≤ 2 2 λ + β , we get
[Equation]
The second set of lemmas presented below describe how R q (Θ t (cid:107) Θ t ) evolves with time in both phases I and II.
Central to our analysis is the following lemma which bounds the rate of change of
Rényi divergence for any pair of diffusion process characterized by their Fokker-Planck equations.
Lemma 29 (Rate of change of Rényi divergence (Chourasia et al., 2021)) .
Let V t , V (cid:48)
t : R d → R d be two time dependent vector ﬁeld such that max
[Equation]
2 ≤ L for all θ ∈ R
[Equation]
and t
[Equation]
For a diffusion process (Θ t ) t ≥ 0 and (Θ (cid:48) t ) t ≥ 0 deﬁned by the Fokker-Planck equations
[Equation]
and
[Equation]
[Equation] respectively, where µ t and µ t are the densities of Θ t and Θ (cid:48) t , the rate of change of Rényi divergence between the two at any t ≥ 0 is upper bounded as
[Equation]
We will apply the above lemma to the Fokker-Planck equation (87) of our pair of tracing diffusion
SDE (84) and solve the resulting differential inequality to prove the bound in (82).
To assist our proof, we rely on the following lemma showing that our two tracing diffusion satisfy the LS inequality described in Deﬁnition G.1, which enables the use the inequality (79) in Lemma 23.
Lemma 30.
If loss (cid:96) ( θ ; x ) is convex and β -smooth, regularizer is r ( θ )
=
λ 2 (cid:107) θ (cid:107) 22 , and learning rate η ≤ 2 2 λ + β , then the tracing diffusion (Θ t )
≤ t ≤ η ( K A + iK ¯A ) and (Θ (cid:48) t ) 0 ≤ t ≤ η ( K A + iK ¯A ) deﬁned in (84) with Θ 0 , Θ (cid:48)
∼ ρ = N (cid:16) 0 , σ 2 λ (1 − ηλ/ 2) I d (cid:17) satisfy LS inequality with constant λ (1 − ηλ/ 2) .
Proof.
For any iteration 0 ≤ k < K A + iK ¯A in the extended run of Noisy-GD, and any 0 ≤ s ≤ η , let’s deﬁne two functions L s , L (cid:48) s : R d → R as follows:
[Equation]
Since r ( · ) is the L 2( λ ) regularizer and (cid:96) ( θ ; x ) is convex and β -smoothness, both L s and L (cid:48) s are λstrongly convex and ( λ + β ) -smooth for all 0 ≤ s ≤ η and any k .
We deﬁne maps T s , T (cid:48)
s : R d → R d as
[Equation]
From a similar argument as in Lemma 28, both T s and T (cid:48) s are (1 − ηλ ) -Lipschitz for learning rate η ≤
[Equation]
Note that the densities of Θ t and Θ (cid:48) t of the tracing diffusion for t = ηk + s can be respectively expressed as
[Equation] [Equation]
where µ ηk and µ (cid:48)
ηk represent the distributions of Θ ηk and Θ (cid:48)
ηk .
We prove the lemma via induction.
Base step: Since Θ 0 , Θ (cid:48) 0 are both Gaussian distributed with variance σ 2 λ (1 − ηλ/ 2) , from Lemmathey satisfy LS inequality with constant λ (1 − ηλ/ 2) .
Induction step: Suppose µ ηk and µ (cid:48)
ηk satisfy LS inequality with constant λ (1 − ηλ/ 2) .
Since equation (95) shows that µ t , µ (cid:48) t are both gaussian convolution on a pushforward distribution of µ ηk , µ (cid:48)
ηk respectively over a Lipschitz function, from Lemma 18 and Lemma 19, both µ t , µ (cid:48) t satisfy
LS inequality with constant
[Equation]
for all ηk ≤ t ≤ η ( k + 1) .
We are now ready to prove the data-deletion bound in (82).
Theorem 31 (Data-Deletion guarantee on (A Noisy-GD , ¯A Noisy-GD ) under convexity) .
Let the weight initialization distribution be ρ = N (cid:16) 0 , σ 2 λ (1 − ηλ/ 2) (cid:17) , the loss function (cid:96) ( θ ; x ) be convex, β -smooth, and L -Lipschitz, the regularizer be r ( θ )
=
λ 2 (cid:107) θ (cid:107) 22 , and learning rate be η < 1 λ + β .
Then Algorithm pair (A , ¯A) satisﬁes a ( q, ε dd ) -data-deletion guarantee under all non-adaptive r -requesters for any noise variance σ 2 > 0 and K A ≥
if
[Equation]
Proof.
Following the preceding discussion, to prove this theorem, it sufﬁces to show that the inequality (82) holds under the stated conditions.
Consider the Fokker-Planck equation described in
Lemma 26 for the pair of tracing diffusions SDEs in (86): at any time t in duration ηk < t ≤ η ( k +1) for any iteration 0 ≤ k < K A + iK ¯A ,
[Equation]
where µ t and µ (cid:48) t are the distribution of Θ t and Θ (cid:48) t .
Since (cid:96) ( θ ; x ) is L -Lipschitz and for any
K A
+ ( i − 1) K ¯ A ≤ k < K A + iK ¯ A
we have D ( k ) [ind] =
D (cid:48)
( k )[ind] , note from the deﬁnition of g k ( θ ) in (86) that
[Equation]
Therefore, applying Lemma 29 to the above pair of Fokker-Planck equations gives that for any t in duration ηk < t ≤ η ( k + 1) ,
[Equation]
Equation (100) suggests a phase change in the dynamics at iteration k = K A
+ ( i − 1) K ¯A .
In phase I, the divergence bound increases with time due to the effect of the differing record in database pairs ( D j , D (cid:48) j ) 0 ≤ j ≤ i − 1 .
In phase II however, the update request u i makes D i − 1 ◦ u i = D
(cid:48)
i − 1 ◦ u i , and so doing gradient descent rapidly shrinks the divergence bound.
This phase change is illustrated in the
Figure 1.
For brevity, we denote R ( q, t ) =
R q ( µ t (cid:107) µ (cid:48) t ) .
Since η < 1 λ + β < 2 2 λ + β , from Lemma 30, the distribution µ (cid:48) t satisﬁes LS inequality with constant λ (1 − λη/ 2) .
So, we can apply Lemma 23 to simplify the above partial differential inequality as follows.
[Equation]
For brevity, let constant c 1 = λ (1 − λη/ 2) and constant c 2 = 2 L 2 σ 2 n 2 .
We deﬁne u ( q, t ) =
R ( q,t ) q .
Then,
[Equation]
For some constant ¯ q > 1 , let q ( s ) =
(¯ q − 1) exp [ c 1 { s − η
( K A + iK ¯A ) } ]
+ 1 and t ( s )
= s .
Note that d q ( s ) d s = c 1 ( q ( s ) − 1) and d t ( s ) d s = 1 .
Therefore, for any ηk < s ≤ η ( k + 1) , the differential inequality followed along the path u ( s )
=
u ( q ( s ) , t ( s )) is
[Equation]
[Equation]
Since the map T k ( · ) in (85) is a differentiable bijection for η < 1 λ + β as per Lemma 28, note that
Lemma 27 implies that lim s → ηk + u ( s )
=
u ( ηk ) .
Therefore, we can directly integrate in the duration
0 ≤ t ≤ η ( K A + iK ¯A ) to get
[Equation]
(Since u (0) =
R ( q (0) , 0) /q
= 0 .)
Noting that q (0) ≥ 1 , on reverting the substitution, we get
[Equation]
(Since η < 1 λ + β )
Recall from our construction that µ η ( K A + iK ¯A ) and µ (cid:48)
η ( K A + iK ¯A ) are the distributions of ¯A( D i − 1 , u i , ˆΘ i − 1 ) and ¯A( D (cid:48)
i − 1 , u i , ˆΘ (cid:48)
i − 1 ) respectively.
Therefore, choosing K ¯A as speciﬁed in the theorem statement concludes the proof.
Our next goal in this section is to provide utility guarantees for the algorithm pair (A Noisy-GD , ¯A Noisy-GD ) in form of excess empirical risk bounds.
For that, we introduce some additional auxiliary results ﬁrst.
The following Lemma 32 shows that excess empirical risks does not increase too much on replacing r records in a database, and Lemma 33 provides a convergence guarantee on the excess empirical risk of Noisy-GD algorithm under convexity.
Lemma 32.
Suppose the loss function (cid:96) ( θ ; x ) is convex, L -Lipschitz, and β -smooth, and the regularizer is r ( θ )
=
λ 2 (cid:107) θ (cid:107) 22 .
Then, the excess empirical risk of any randomly distributed parameter Θ for any database D ∈ X n after applying any edit request u ∈ U r that modifes no more than r records is bounded as
[Equation] [Equation]
Proof.
Let θ ∗D and θ ∗D◦
u be the minimizers of objectives L D ( · ) and L D◦ u ( · ) as deﬁned in ( ?? ).
From λ -strong convexity of the L D ,
[Equation]
[Equation] From optimality of θ ∗D◦ u and L -Lipschitzness of (cid:96) ( θ ; x ) , we have
[Equation]
Combining the two inequalities give
[Equation]
Therefore, from ( λ + β ) -smoothness of L D◦ u and λ -strong convexity of L D , we have
[Equation]
Lemma 33 (Accuracy of Noisy-GD) .
For convex, L -Lipschitz, and, β -smooth loss function (cid:96) ( θ ; x ) and regularizer r ( θ ) = λ 2 (cid:107) θ (cid:107) 22 , if learning rate η < 1 λ + β , the excess empirical risk of Θ ηK =
Noisy-GD ( D , Θ 0 , K ) for any D ∈ X n is bounded as
[Equation]
Proof.
Let Θ ηk denote the k th iteration parameter of Noisy-GD run.
Recall that k + 1 th noisy gradient update step is
[Equation]
From ( β + λ ) -smoothness of L D , we have
[Equation]
On taking expectation over the joint distribution of Θ ηk , Θ η ( k +1) , Z k , the above simpliﬁes to
[Equation]
Let θ ∗D = arg min θ ∈ R d L D ( θ ) .
From λ -strong convexity of L D , for any θ ∈ R d , we have
[Equation]
[Equation]
Let γ = λη (2 − η
( λ + β )) .
Plugging this in the above inequality, and substracting L D ( θ ∗D ) on both sides, for η < 1 λ + β , we get
E (cid:2)
L
[Equation]
For η < 1 λ + β , note that γ ≥ λη , and so
[Equation]
Finally, we are ready to prove our main Theorem 6 showing that algorithm pair (A Noisy-GD , ¯A Noisy-GD ) solves the data-deletion problem as described in Section 4.
We basically combine the RDP guarantee in Theorem 25, non-adaptive data-deletion guarantee in Theorem 31, and prove excess empirical risk bound using Lemma 33 and Lemma 32.
Theorem 6 (Accuracy, privacy, deletion, and computation tradeoffs) .
Let constants λ, β, L > 0 , q > 1 , and 0 < ε dd ≤ ε dp .
Deﬁne constant κ = λ + βλ .
Let the loss function (cid:96) ( θ ; x ) be twice differentiable, convex, L -Lipschitz, and β -smooth, the regularizer be r ( θ )
=
λ 2 (cid:107) θ (cid:107) 22 .
If the learning rate be η = 1 2( λ
[Equation] ) , the gradient noise variance is σ 2 = 4 qL [Equation] n 2 , and the weight initialization distribution is ρ = N (cid:16) 0 ,
[Equation]
, then
both A Noisy-GD and ¯A Noisy-GD are ( q, ε dp ) -RDP for any K A , K ¯A ≥ 0 , (2.)
pair (A Noisy-GD , ¯A Noisy-GD ) satisﬁes ( q, ε dd ) -data-deletion all non-adaptive r -requesters
if
[Equation]
(112) (3.) and all models in (ˆΘ i ) i ≥ 0 produced by (A Noisy-GD , ¯A Noisy-GD , Q ) on any D 0 ∈ X n , where
Q is any r -requester, have an excess empirical risk err(ˆΘ i ; D i ) =
O (cid:16) qd ε dp n 2 (cid:17) if K A ≥ 4 κ log (cid:18) ε dp n4 qd (cid:19) , and K ¯A ≥ 4 κ log max (cid:26) 5 κ, 8 ε dp rqd (cid:27) .
Proof.
Privacy.
By Theorem 25, the Noisy-GD with K iterations will be ( q, ε dp ) -RDP for the stated choice of loss function, regularizer, and learning rate as long as σ 2 ≥ 4 qL 2 λε dp n 2 (cid:0) 1 − e − ληK/ 2 (cid:1) .
Therefore, if we set σ 2 = 4 qL 2 λε dp n 2 , Noisy-GD is ( q, ε dp ) -RDP for any K .
For the same σ 2 , both
A Noisy-GD and ¯A Noisy-GD are also ( q, ε dp ) -RDP for any K A and K ¯A as they run Noisy-GD on respective databases for generating the output.
(2.) Deletion.
By Theorem 31, for the stated choice of loss function, regularizer, learning rate, and weight initialization distribution, the algorithm pair (A Noisy-GD , ¯A Noisy-GD ) satisﬁes ( q, ε dd ) - data-deletion under all non-adaptive r -requesters Q if K ¯A ≥ 2 ηλ log (cid:16) 4 qL 2 λε dd σ 2 n 2 (cid:17) .
By plugging in σ 2 = 4 qL 2 λε dp n 2 and η = 1 2( λ + β ) , this constraint simpliﬁes to K ¯A ≥ 4 κ log ε dp ε dd .
Accuracy.
We prove the induction hypothesis that under the conditions stated in the theorem, err(ˆΘ i ; D i ) ≤ 10 κqdL 2 λε dp n 2 for all i ≥ 0 .
Base case: The minimizer θ ∗D 0 of L D 0 satisﬁes
[Equation]
As a result, the excess empirical risk of initialization weights Θ 0 ∼ ρ = N (cid:16) 0 ,
[Equation]
on L D
[Equation]
is bounded as
[Equation]
Since ˆΘ 0 =
A Noisy-GD ( D 0 ) =
Noisy-GD ( D 0 , Θ 0 , K A ) , by
Lemma 33, running
K A ≥ 2 κ log (cid:16) ε dp n 2 4 qd (cid:17) iterations gives
[Equation]
(On substituting
[Equation]
[Equation]
(Since K A ≥ 4 κ log (cid:16) ε dp n 2 4 qd (cid:17) )
Induction step: Assume that err(ˆΘ i − 1 ; D i − 1 ) ≤ 10 κqdL 2 λε dp n 2 .
Since ˆΘ i = ¯A Noisy-GD ( D i − 1 , u i , ˆΘ i − 1 ) =
Noisy-GD ( D i , ˆΘ i − 1 , K ¯A ) , by Lemma 33 and Lemma 32, running K ¯A ≥ 2 κ log max (cid:110) 5 κ, 8 r 2 qd (cid:111) iterations gives
[Equation]
dp n(Since K ¯A ≥ 4 κ log(5 κ ) ) ≤ 10 κqdLλε dp n(Since K ¯A ≥ 4 κ log 8 ε dp r 2 qd )
G. P ROOFS FOR S UBSECTION 5.
In this Appendix, we provide a proof of our data-deletion and utility guarantee in Theorem 7 which applies to non-convex but bounded losses (cid:96) ( θ ; x ) under L 2 regularizer r ( θ ) .
Suppose D 0 ∈ X n is an arbitrary database, Q is any non-adaptive r -requester, and (ˆΘ i ) i ≥ 0 is the model sequence generated by the interaction of (A Noisy-GD , ¯A Noisy-GD , Q ) .
Our ﬁrst goal will be to prove ( q, ε dd ) -data deletion guarantee on (A Noisy-GD , ¯A Noisy-GD ) and we will later use it for arguing utility as well.
Recall from
Deﬁnition 4.1 that to prove ( q, ε dd ) -data-deletion, we need to construct a map π Q i : X n → O such that for all i ≥ 1 and any u i [-− 
-]∈ U r ,
[Equation]
for all (cid:104) ind [Equation] u i [Equation]
Our construction of π ui for this proof is completely different from the one described in Appendix G.4.
As discussed in Remark 3, since Q is non-adaptive, it sufﬁces to show that there exists a map π : X n → O such that for all i ≥ 1 , [Equation]
dd [Equation] (116) for all D 0 ∈ X n and all edit sequences ( u i ) i ≥ 1 from U r .
Our mapping of choice for the purpose is the Gibbs distribution with the following density:
[Equation]
[Equation]
The high-level intuition for this construction is that Noisy-GD can be interpreted as Unadjusted
Langevin Algorithm (ULA) (Roberts & Tweedie, 1996), which is a discretization of the Langevin diffusion (described in eqn.
that eventually converges to this Gibbs distribution (see Appendix G. for a quick refresher).
However, showing a convergence for ULA (in indistinguishability notions like
Rényi divergence) to this Gibbs distribution, especially in form of non-asymptotic bounds on the mixing time and discretization error has been a long-standing open problem.
Recent breakthrough results by Vempala & Wibisono (2019) followed by Chewi et al.
(2021) resolved this problem with an elegant argument, relying solely on isoperimetric assumptions over (117) that hold for non-convex losses.
Our data-deletion argument leverages this rapid convergence result to basically show that once Noisy-GD reaches near-indistinguishability to its Gibbs mixing distribution, maintaining indistinguishability to subsequent Gibbs distribution corresponding to database edits require much fewer
Noisy-GD iterations than fresh retraining (i.e. data deletion is faster than retraining).
We start by presenting Chewi et al.
(2021)’s convegence argument adapted to our Noisy-GD formulation, with a slighlty tighter analysis that results in a log( q ) improvement in the discretization error over the original.
Consider the discrete stochastic process (Θ ηk ) 0 ≤ k ≤ K induced by parameter update step in Noisy-GD algorithm when run for K iterations on a database D with an arbitrary start distribution Θ 0 ∼ µ 0 .
We interpolate each discrete update from Θ ηk to Θ η ( k +1) via a diffusion process Θ t deﬁned over time ηk ≤ t ≤ η ( k + 1) as [Equation]
[Equation]
where Z t is a Weiner process.
Note that if Θ ηk models the parameter distribution after the k th update, then Θ η ( k +1) models the parameter distribution after the k + 1 th update.
On repeating this construction for all k = 0 , · · · , K , we get a tracing diffusion { Θ t } t ≥ 0 for Noisy-GD (which is different from (84)).
We denote the distribution of random variable Θ t with µ t .
The tracing diffusion during the duration ηk ≤ t ≤ η ( k + 1) is characterized by the following Fokker-Planck equation.
Lemma 34 (Proposition 14 (Chewi et al., 2021)) .
For tracing diffusion Θ t deﬁned in (118) , the equivalent Fokker-Planck equation in the interval ηk ≤ t ≤ η ( k + 1) is
[Equation]
where π ( D ) is the Gibbs distribution deﬁned in (117) .
Proof.
Conditioned on observing parameter Θ ηk
= θ ηk , the process (Θ t ) ηk ≤ t ≤ η ( k +1) is a Langevin diffusion along a constant Vector ﬁeld ∇L D ( θ ηk ) .
Therefore, the conditional probability density µ t | ηk ( ·| θ ηk ) of Θ t given θ ηk follows the following Fokker-Planck equation.
[Equation]
Taking expectation over Θ ηk , we have
[Equation]
where µ ηk | t is the conditional density of Θ ηk given Θ t .
For the last equality, we have used the fact that ∇L D = − σ 2 ∇ log π ( D ) from (117).
The following lemma provides a partial differential inequality that bounds the rate of change in Rényi divergence R q ( µ t (cid:107) π ( D )) using Fokker-Planck equation (119) of Noisy GD’s tracing diffusion.
Lemma 35 (Proposition 15 (Chewi et al., 2021)) .
Let ρ t := µ t / π ( D ) where π ( D ) is the Gibbs distribution deﬁned in (117) and ψ t :
= ρ q − 1 t / E q ( ρ t (cid:107) π ( D )) .
The rate of change in R q ( µ t (cid:107) π ( D )) along racing diffusion in time ηk ≤ t ≤ η ( k + 1) is bounded as
[Equation]
Proof.
For brevity
[Equation]
let ∆ t ( · ) =
E [ ∇L D (Θ ηk ) − ∇L D (Θ t ) | Θ t = · ] in context of this proof.
From
Lebinz integral rule
[Equation]
we have
[Equation]
Note that the expectation in ∆ t ( · ) is over the conditional distribution µ ηk | t while the expectation in
F 1 is over µ t .
Therefore, we can combine the two to get an expectation over the unconditional joint distribution over Θ t and Θ ηk as follows.
[Equation]
Substituting it in the preceding inequality proves the proposition.
We need to solve the PDI (121) to get a convergence bound for Noisy-GD.
To help in that, we ﬁrst introduce the change of measure inequalities shown in Chewi et al.
Lemma 36 (Change of measure inequality (Chewi et al., 2021)) .
If (cid:96) ( θ ; x ) is β -smooth, and regularizer is r ( θ )
= λ 2 (cid:107) θ (cid:107) 22 , then for any probability density µ on R d ,
[Equation]
where π ( D ) is the Gibbs distribution deﬁned in (117) .
Proof.
Consider the Langevin diffusion (63) described in Appendix G.1 over the potential L D .
The
Gibbs distribution π ( D ) is its stationary distribution, and the diffusion’s inﬁntesimal generator G applied on the L D gives
[Equation]
Therefore,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Another change in measure inequality needed for the proof is the Donkser-Vardhan variational principle.
Lemma 37 (Donsker-Vardhan Variational principle (Donsker & Varadhan, 1983)) .
If ν and ν (cid:48) are two distributions on R d such that ν (cid:28) ν (cid:48) , then for all functions f : R d → R ,
[Equation] [Equation]
We are now ready to prove the rate of convergence guarantee for Noisy-GD following Chewi et al.
(2021)’s method, but with a more reﬁned analysis that leads to a improvement of log q factor in the discretization error (compared to the original (Chewi et al., 2021, Theorem 4)).
Theorem 38 (Convergence of Noisy-GD in Rényi divergence) .
Let constants β, λ, σ 2 ≥ 0 and q, B > 1 .
Suppose the loss function (cid:96) ( θ ; x ) is ( σ 2 log( B ) / 4) -bounded and β -smooth, and regularizer is r ( θ )
=
λ 2 (cid:107) θ (cid:107) 2 2 .
If step size is η ≤ λ 64 Bq 2 ( β + λ ) 2 , then for any database D ∈ X n and any weight initialization distribution µ 0 for Θ 0 , the Rényi divergence of distribution µ ηK of output model Θ ηK =
Noisy-GD ( D , Θ 0 , K ) with respect to the Gibbs distribution π ( D ) deﬁned in (117) shrinks as follows:
[Equation]
Proof.
From ( β + λ ) -smoothness of loss L D we have that for any ηk ≤ t ≤ η ( k + 1) ,
[Equation]
Let ρ t := µ t π ( D ) and ψ t :
=
ρ q − 1 t / E q ( ρ t (cid:107) π
( D )) .
If η ≤ 1 2 √ 2( β + λ ) , we rearrange to get the following and use it to get the following bound on the discretization error in (121):
[Equation]
Hence, for solving the PDI (121), we have to bound the three expectations F 1 and F 2 .
Bounding F 1 .
Note that
[Equation]
So, ψ t µ t ( θ ) :
= ψ t ( θ ) µ t ( θ ) is a probablitly density function on R d .
On applying the measure change Lemma 36 on it, we get
[Equation]
Bounding F 2 .
Since ψ t µ t is a valid density on R d , the joint density ψ t µ t,z ( θ, z ) :
= ψ t ( θ ) µ t,z ( θ, z ) where µ t,z is the joint density of Θ t and Z k is also a valid density.
Note that the F 2 is an expectation on (cid:107) Z k (cid:107) 22 taken over the joint density ψ t µ t,z .
We can perform a measure change operation using Donsker-Vardhan principle to get
[Equation]
where we simpliﬁed the second term using the fact that the marginal µ z of µ t,z is a standard normal Gaussian.
The random variable (cid:107) Z k (cid:107) 22 is distributed according to the Chi-squared distribution χ 2 d with d degrees of freedom.
Since the moment generating function of
Chi-squared distribution is M χ 2 d ( t ) =
E
X ∼ χ 2 d
[exp( tX )]
= (1 − 2 t ) − d/ 2 for t < 12 , we can simplify the second term in F 2 as
[Equation]
The KL divergence term can be simpliﬁed as follows.
[Equation]
marginalization
[Equation] Since R q ( µ t (cid:107) π ( D )) > 0 )
Note that under the assumptions of the Theorem, π ( D ) satisﬁes log-Sobolev inequality (74) with constant λ/B (i.e. satisﬁes LS( λ/B ) ).
To see this, recall from Lemma
that the
Gaussian distribution ρ ( θ ) =
N (cid:16) 0 , σ 2 λ I d (cid:17) satisﬁes LS( λ ) inequality.
Since loss (cid:96) ( θ ; x ) is ( σ 2 log( B ) / 4) -bounded, the density ratio π ( D )( θ ) ρ ( θ ) ∈ (cid:104) 1 √ B , √ B (cid:105) .
The claim therefore follows from Lemma 20.
Using this inequality, from Lemma 21 we have
[Equation]
On combining all the two bounds on F 1 and F 2 and rearranging, we have
[Equation]
Let step size be η ≤ min (cid:110)
Bλ ,
[Equation]
(cid:111) .
Then, the ﬁrst term above is bounded as
[Equation]
Let η ≤ 1 4( β + λ ) .
Then, in the third term, ( η ( β + λ ) + log 2) ≤ 1 .
Plugging the bound on discretization error back in the PDI (121), we get
[Equation]
[Equation]
Since π ( D ) satisﬁes LS( λ/B ) inequality, from Lemma 23 this PDI reduces to
[Equation]
Let c 1 = λ 2 B and c 2 = 16 dη ( β + λ ) 2 .
Additionally, let u ( q, t ) =
R q ( µ t (cid:107) π
( D ))
q .
Then, ∂ t R q ( µ t (cid:107) π ( D )) + c(cid:18)
[Equation]
q
[Equation]
t
[Equation]
D
[Equation]
q
[Equation]
q = ⇒ ∂
t R q ( µ t (cid:107) π
( D ))
[Equation]
q
[Equation]
= ⇒ ∂ t u ( q, t )
+ c 1 u ( q, t ) + c 1 ( q − 1) ∂ q u ( q, t ) ≤ c 2 .
For some constant ¯ q ≥ 1 , let q ( s ) =
(¯ q − 1) exp( c 1 ( s − ηK ))
+ 1 , and t ( s )
= s .
Note that d q ( s ) d s = c 1 ( q ( s ) − 1) and d t ( s ) d s = 1 .
Therefore, for any 0 ≤ t ≤ ηK , the PDI above implies the following differential inequality is followed along the path u ( s )
=
u ( q ( s ) , t ( s )) .
[Equation]
On reversing the parameterization of q and t , we get
[Equation]
Since q (0) > 1 and ¯ q = q ( ηK ) > q (0) , from monotonicity of Rényi divergence in q , we get
[Equation]
[Equation]
Finally, noting that for constants
B, q > and β, λ ≥
[Equation]
step size
[Equation]
completes the proof.
We will use Theorem 38 for proving the data-deletion and utility gaurantee on (A Noisy-GD , ¯A Noisy-GD ) .
We need the following result that shows that Gibbs distributions enjoy strong indistinguishability on bounded perturbations to its potential function (which is basically why the exponential mechanism satisﬁes ( ε, 0) -DP
(Wang et al., 2015; Dwork et al., 2014)).
Lemma 39 (Indistinguishability under bounded perturbations) .
For two potential functions L , L (cid:48) :
R d → R and some constant σ 2 , let ν ∝ e −L /σ 2 and ν (cid:48)
∝ e −L (cid:48) /σ 2 be the respective Gibbs distributions.
If |L ( θ ) − L (cid:48) ( θ ) | ≤ c for all θ ∈ R d , then R q ( ν (cid:107) ν (cid:48) )
≤ 2 cσ 2 for all q > 1 .
Proof.
The Gibbs distributions ν , ν (cid:48) have a density
[Equation]
and
[Equation]
where Λ , Λ (cid:48) are the respective normalization constants.
If for all θ ∈ R d , the potential difference |L
[Equation]
θ ) − L (cid:48)
( θ ) | ≤ c , then
[Equation]
In Theorem 7, we show that (A Noisy-GD , ¯A Noisy-GD ) solves the data-deletion problem described in
Section 4 even for non-convex losses.
Our proof uses the convergence Theorem 38 and indistinguishability for bounded perturbation Lemma 39 to show that the data-deletion algorithm ¯A Noisy-GD can consistently produce models indistinguishable to the corresponding Gibbs distribution (117) in the online setting at a fraction of computation cost of A Noisy-GD .
As discussed in Remark 3, such an indistinguishability is sufﬁcient for ensuring data-deletion for non-adaptive requests.
As for adaptive requests, the well-known RDP guarantee of Abadi et al.
combined with our reduction
Theorem 5 offers a data-deletion guarantee for (A Noisy-GD , ¯A Noisy-GD ) under adaptivity.
Our proof of accuracy for the data-deleted models leverages the fact that Gibbs distribution (117) is an almost excess risk minimizer as shown in the following Theorem 40.
Since our data-deletion guarantee is based on near-indistinguishability to (117), this property also ensures near-optimal excess risk of data-deleted models.
Theorem 40 (Near optimality of Gibbs sampling) .
If the loss function (cid:96) ( θ ; x ) is σ 2 log( B ) / 4bounded and β -smooth, the regularizer is r ( θ ) =
λ 2 (cid:107) θ (cid:107) 22 , then the excess empirical risk for a model ¯Θ sampled from the Gibbs distribution π ( D ) ∝ e −L D /σ 2 is
[Equation]
Proof.
We simplify expected loss as
[Equation]
where
[Equation]
is the differential entropy of π ( D ) , and Λ D = (cid:82) e −L D /σ
2 d θ is the normalization constant.
Since the potential function L D is ( λ + β ) -smooth, we have
[Equation]
Since (cid:96) ( θ ; x ) is σ 2 log( B ) / 4 -bounded, note that for the Gaussian distribution ρ ∼ N (cid:16) 0 , σ 2 λ I d (cid:17) , the density ratio lies in π ( D )( θ ) ρ ( θ ) ∈ (cid:104) 1 √ B , √ B (cid:105) for all θ ∈ R d .
We decompose entropy H ( π ( D )) into cross-entropy and KL divergence to get
[Equation]
Since KL
[Equation] [Equation]
Since
[Equation]
D
[Equation]
On combining the bounds, we get
[Equation]
Theorem 7 (Accuracy, privacy, deletion, and computation tradeoffs) .
Let constants λ, β, L, σ 2 , η > 0 , q, B > 1 , and 0 < ε dd ≤ ε dp < d .
Let the loss function (cid:96) ( θ ; x ) be σ 2 log( B )-bounded, L -Lipschitz and β -smooth, the regularizer be r ( θ ) =
λ 2 (cid:107) θ (cid:107) 22 , and the weight initialization distribution be ρ = N (cid:16) 0 , σ 2 λ I d (cid:17) .
Then,
both A Noisy-GD and ¯A Noisy-GD are ( q, ε dp ) -RDP for any η ≥ 0 and any K A , K ¯A ≥
if
[Equation]
(2.) pair (A Noisy-GD , ¯A Noisy-GD ) satisfy ( q, ε dd ) -data-deletion under all non-adaptive r -requesters for any σ 2 > 0 , if learning rate is η ≤
[Equation]
( β + λ ) 2 and number of iterations satisfy
[Equation]
(3.) and all models in sequence (ˆΘ i ) i ≥ 0 output by (A Noisy-GD , ¯A Noisy-GD , Q ) on any D 0 ∈ X n , where Q is an r -requester, satisfy err(ˆΘ i ; D i ) =
˜
O
[Equation]
inequalities in (136) and (135) are equalities.
Proof.
Privacy.
By Theorem 24, Noisy-GD with K iterations on an L -Lipschitz loss function satisﬁes ( q, ε dp ) -RDP for any initial weight distribution ρ and learning rate η ≥ 0 if σ 2 = qL 2 ε dp n 2 · ηK .
Since, both A Noisy-GD and ¯A Noisy-GD run Noisy-GD for K A and K ¯A iterations respectively, setting the noise variance given in the Theorem statement ensures ( q, ε dp ) -RDP for both.
Deletion.
For showing data-deletion under non -adaptive requests, recall that
it is sufﬁcient to show that there exists a map π : X n → O such that for all
i ≥ 1 ,
[Equation]
for all edit sequences ( u i ) i ≥ 1 from U r , where (ˆΘ i ) i ≥ 0 is the sequence of models generated by the interaction of (A Noisy-GD , ¯A Noisy-GD , Q ) on any database D 0 ∈ X n .
For all i ≥ 0 , let ˆ µ
i denote the distribution of ˆΘ i .
We prove (137) via induction.
Base step: Note that the initial weight distribution ρ = N (cid:16) 0 , σ 2 λ I d (cid:17) has a density proportional to e − r
( θ ) /σ 2 and the distribution π ( D 0 ) has a density proportional to e −L D
( θ ) /σ 2 .
Since both of these are Gibbs distributions with their potential difference |L D
( θ ) − r ( θ ) | ≤ σ 2 log( B ) / 4 for all θ ∈ R d due to boundedness assumption on (cid:96) ( θ ; x ) , we have from Lemma 39 that
[Equation]
[Equation]
Under the stated assumptions on loss (cid:96) ( θ ; x ) and learning rate η , note that the convergence Theorem 38 holds.
Since ˆΘ 0 =
A Noisy-GD ( D 0 ) =
Noisy-GD ( D 0 , Θ 0 , K A ) , where Θ 0 ∼ ρ , we have
[Equation]
(cid:18) log( B ) (cid:19) + ε dd (Since η ≤ λε dd 64 dqB ( β + λ ) 2 ) ≤ ε dd (Since K A ≥ 2 Bλη log (cid:16) q log( B ) ε dd (cid:17) )
Induction step: Suppose R q (ˆ µ i − 1 (cid:107) π ( D i − 1 )) ≤ ε dd .
Again, from boundedness of (cid:96) ( θ ; x ) , we have |L D i − 1 ( θ ) − L D
i ( θ ) | ≤ rσ 2 log B 2 n for all θ ∈ R d .
Therefore, from Lemma 39 we have for all q >that
[Equation]
[Equation]
So from the weak triangle inequality Theorem 17 of Rényi divergence,
[Equation]
Note that
K ¯A ≥ K A − 2 Bλη log
[Equation]
(cid:19) .
Since ˆΘ i = ¯A Noisy-GD ( D i − 1 , u i , ˆΘ i − 1 ) =
Noisy-GD ( D i , ˆΘ i − 1 , K ¯A ) , from convergence Theorem 38 we have
[Equation]
(From (140) and constraint η ≤ [Equation] dqB (
[Equation]
+
[Equation]
≤ ε dd . (Since K ¯A ≥ 2 Bλη log
[Equation]
dd
[Equation]
log
[Equation]
B
[Equation]
Hence, by induction, R q (ˆ µ i (cid:107) π
( D i )) ≤ ε dd holds for all i ≥ 0 .
Accuracy.
Let θ ∗D i =
arg min θ
∈ R
d L D i ( θ ) , and ¯Θ i ∼ π ( D i ) .
We decompose the excess empirical risk of Noisy-GD as follows: [Equation]
[Equation]
The second term is the suboptimality of Gibbs distribution and by Theorem 40, it is bounded as
[Equation]
Due to L -Lipschitzness and ( λ + β ) -smoothness of L D i , for any coupling Π of ˆΘ i and ¯Θ i , the ﬁrst term is bounded as
[Equation]
Jensen’s inequality)
Recall that the distribution π ( D ) satisﬁes LS( λ/B ) inequality.
On choosing the coupling Π to be the inﬁmum [Equation] we get the following bound on Wasserstein’s distance from Lemma 22.
[Equation]
The last inequality above follows from monotonicity of Rényi divergence in q and the fact that lim q → 1 R q ( ν (cid:107) ν (cid:48) )
= KL ( ν (cid:107) ν (cid:48) ) .
Therefore, on combining all the bounds we get err
[Equation]
ˆΘ; D ) ≤ Lσ
[Equation]
Note that if the constraints on K A and K ¯A in (136) and on σ 2 in (135) are equalities instead, we have [Equation]
[Equation]
where ˜ O ( · ) hides logarithmic factors.
Therefore, the excess empirical risk has an order
[Equation]