K NOWLEDGE R EMOVAL IN S AMPLING - BASED B AYESIAN I NFERENCE
A BSTRACT
The right to be forgotten has been legislated in many countries, but its enforcement in the AI industry would cause unbearable costs.
When single data deletion requests come, companies may need to delete the whole models learned with massive resources.
Existing works propose methods to remove knowledge learned from data for explicitly parameterized models, which however are not appliable to the sampling-based Bayesian inference, i.e. , Markov chain Monte
Carlo (MCMC), as MCMC can only infer implicit distributions.
In this paper, we propose the ﬁrst machine unlearning algorithm for MCMC.
We ﬁrst convert the MCMC unlearning problem into an explicit optimization problem.
Based on this problem conversion, an MCMC inﬂuence function is designed to provably characterize the learned knowledge from data, which then delivers the MCMC unlearning algorithm.
Theoretical analysis shows that MCMC unlearning would not compromise the generalizability of the MCMC models.
Experiments on Gaussian mixture models and Bayesian neural networks conﬁrm the effectiveness of the proposed algorithm.
The code is available at https: //github.com/fshp971/mcmc-unlearning .
I NTRODUCTION
“The right to be forgotten” refers to the right of individuals to request data controllers such as tech giants to delete the data collected from them.
It has been recognized in many countries through legislation, including the European Union’s General Data Protection Regulation (2016) and the California Consumer Privacy Act (2018).
In the ﬁeld of machine learning, legal experts suggest that companies may need to delete and re-train the machine learning models to meet the legal requirements of the right to be forgotten (Li et al., 2018;
Garg et al., 2020).
However, training a machine learning model would usually cost massive amounts of resources, including data, energy, and time.
Thus, the enforcement of the right to be forgotten in the AI industry may result in unbearable costs.
Toward this end, recent works (Cao & Yang, 2015; Guo et al., 2020) propose machine unlearning algorithms to efﬁciently characterize and remove the knowledge learned from speciﬁc data.
To enable knowledge removal analysis, existing unlearning approaches usually require the machine learning models to be explicitly parameterized, which however do not cover an important family of machine learning algorithm, the Markov chain Monte Carlo (MCMC).
MCMC is a sampling-based
Bayesian inference method that aims to draw samples along a Markov chain to approximate a target posterior distribution (Hastings, 1970; Geman & Geman, 1984; Welling & Teh, 2011).
Distributions inferred by MCMC are usually implicitly parameterized, which makes it difﬁcult to directly apply existing unlearning algorithms to the MCMC unlearning problem.
In this paper, we propose an MCMC unlearning algorithm to realize the right to be forgotten in the sampling-based Bayesian inference for the ﬁrst time.
The key of tackling the MCMC unlearning problem is to design methods to directly analyze the implicit parameter of the learned distribution.
To this end, we propose to employ the learned implicit distribution to approximate the target posterior under the measure of KL-divergence, which then converts the MCMC unlearning problem to an optimization problem with an explicit distribution parameter for knowledge removal analysis.
Based on this conversion, an MCMC inﬂuence function is designed (Huber, 2004; Koh & Liang, 2017) to characterize the knowledge learned from single data samples.
The new inﬂuence function then delivers the MCMC unlearning algorithm, which removes data knowledge by directly subtracting the corresponding inﬂuence from the learned distribution.
Theoretical analyses are conducted for the proposed MCMC unlearning algorithm.
For the knowledge removal analysis, we prove that the proposed algorithm can provably realize a ε -knowledge removal, which is a new notation deﬁned for evaluating the machine unlearning performance in
Bayesian inference.
For the generalization analysis, a PAC-Bayesian generalization upper bound (McAllester, 1998; 1999; 2003; He et al., 2019) is established for the distribution processed by the MCMC unlearning algorithm.
The upper bound shows that the difference introduced by MCMC unlearning in the generalization upper bound is no larger than the order of O ( (cid:112) | S (cid:48) | /N ) .
This result demonstrates that the proposed algorithm would not compromise the generalization ability of the learned distribution.
In summary, our work has four main contributions: (1) We enable the knowledge removal analysis in
MCMC by converting the MCMC unlearning problem to an optimization problem.
(2) Based on this problem conversion, we design an MCMC inﬂuence function to characterize the knowledge learned from data, which then delivers the MCMC unlearning algorithm.
(3) We conduct theoretical analyses and prove that the MCMC unlearning can realize an ε -knowledge removal and has little impact on the generalization ability of the learned distribution.
(4) We conduct comprehensive experiments to verify the effectiveness of the proposed algorithm, which include Gaussian mixture models for clustering on synthetic data, and Bayesian neural networks for classiﬁcation on real-world dataset.
The results suggest that the MCMC unlearning can effectively remove the knowledge learned from the requested data while retaining the information learned from other data intact.
R ELATED W ORKS
Markov chain Monte Carlo.
MCMC is a Bayesian inference method that would draw samples along a Markov chain, in which the drawn samples are proved to converge to that from a targeted distribution (Hastings, 1970).
Although many improvements have been made such as Hamiltonian
Monte Carlo (HMC) (Duane et al., 1987; Neal et al., 2011), Gibbs sampling (Geman & Geman, 1984; George & McCulloch, 1993) and slice sampling (Neal, 2003; Walker, 2007), MCMC still suffers from huge computational cost when inferring on large-scale dataset, e.g. in computer vision and image processing tasks (Zhang et al., 2020b; Wang et al., 2020), since each sampling step in
MCMC requires computation over the whole dataset.
To tackle this issue, Welling & Teh (2011) introduce a scalable MCMC sampler named stochastic gradient Langevin dynamics (SGLD) that by adding a proper noise to a standard stochastic gradient optimization algorithm (Robbins & Monro, 1951), which can help the update iterations converge to samples from the true posterior distribution.
Since that, a variety of stochastic gradient MCMC samplers (SG-MCMC) have been proposed, including stochastic gradient HMC (SGHMC) (Chen et al., 2014), Patterson & Teh (2013), Ahn et al.
(2012), Ding et al.
(2014), and Zhang et al.
(2020a).
Ma et al.
(2015) further design a complete framework for constructing SG-MCMC.
Machine unlearning.
Machine unlearning aims to remove the knowledge learned from speciﬁc data in efﬁcient manners (Cao & Yang, 2015).
Existing approaches can be roughly divided into exact methods and approximate methods .
Exact methods aim to recover the model trained without the removed data exactly and are usually realized via reducing the retraining cost.
For example, Ginart et al.
(2019) design an efﬁcient unlearning method for k -means clustering via model quantization.
Bourtoule et al.
(2021) propose a SISA framework that would only retrain the affected part of the model during data deletion.
Other advances include Ullah et al.
and Brophy & Lowd (2021).
On the other hand, approximate methods aim to modify the trained model to approximate that trained only on the remaining data.
A series of works (Guo et al., 2020; Golatkar et al., 2020; 2021) are designed via characterizing the inﬂuence of data on the trained model with inﬂuence functions (Cook & Weisberg, 1982;
Huber, 2004; Koh & Liang, 2017).
Other approaches include Baumhauer et al.
(2020) and Izzo et al.
Some works have studied machine unlearning in Bayesian inference (Nguyen et al., 2020;
Gong et al., 2021).
However, these works only focus on variational inference, an optimization-based Bayesian inference method, and could not be applied to samplingbased Bayesian inference, MCMC.
In contrast, our approach is the ﬁrst machine unlearning method for MCMC, with a theoretical knowledge removal guarantee.
P RELIMINARIES
Suppose p ( z | θ ) is a parameterized distribution over the sample space Z , where θ ∈ Θ ⊂ R d is the parameter with a prior distribution p ( θ ) .
Suppose S = { z 1 , z 2 , · · · , z N } is a dataset consists of N samples, where each sample z
i ∈ Z is drawn from the parameterized distribution p ( z | θ ) , i.e. , z i ∼ p ( z | θ ) .
Bayesian inference aims to infer the posterior distribution p ( θ | S ) of the parameter θ ,
[Equation]
For simplicity, for the given dataset S , we let p S denotes the true posterior distribution p
( θ | S ) , and ˆ p S denotes the posterior distribution that inferred by Bayesian inference.
Usually, the denominator of the posterior p S has no closed-form solution, which barriers directly calculating the posterior distribution.
Thereby, researchers have designed methods to approximate the target posterior.
A canonical approach is the sampling-based Bayesian inference method, Markov chain Monte Carlo (MCMC) (Hastings, 1970).
MCMC infers a posterior distribution via ﬁrst constructs a Markov chain and then draws samples according to the state of the chain.
When the target posterior is the stationary distribution of the
Markov chain, it is proved that the drawn samples will converge to that from the target posterior.
However, MCMC methods usually suffer from scalable issues on large-scale datasets.
To this end, stochastic gradient MCMC (SG-MCMC) (Ma et al., 2015) leverage mini-batch estimation (Robbins & Monro, 1951) to enable scalable sampling.
In SG-MCMC, the posterior distribution p ( θ | S ) can be rewritten as p ( θ | S )
∝ exp( − U ( θ )) , where U ( θ ) is the potential function deﬁned as
U ( θ ) = − (cid:80) z i ∈ S log p ( z i | θ ) − log p ( θ ) .
In each sampling step, drawing samples with SG-MCMC requires one to estimate the potential function U ( θ ) with mini-batch data ˜ S ⊂ S as follows,
[Equation]
Typical SG-MCMC methods include SGLD and SGHMC.
Stochastic gradient Langevin dynamics (SGLD)
(Welling & Teh, 2011) inserts Gaussian noise to stochastic gradients.
In each sampling step, SGLD updates the posterior sample θ t as follows,
[Equation]
where W is drawn from the standard Gaussian distribution N (0 , I ) and η t > 0 is the step size.
To improve the sampling efﬁciency of SGLD, stochastic gradient Hamiltonian Monte Carlo (SGHMC) (Chen et al., 2014) inserts a momentum v t into the posterior sample update as below,
[Equation]
where W is also drawn from the Gaussian distribution N (0 , I ) , η t > 0 is the step size, and α ∈ (0 , 1) is the momentum factor.
The initial momentum term v 0 is drawn from the Gaussian distribution N (0 , η 0 I ) .
To ensure the drawn samples converge to the targeted posterior, the step size η t for both SGLD and
SGHMC needs to satisfy (Welling & Teh, 2011): (1) (cid:80) ∞
t =1 η t = ∞ ; and (2) (cid:80) ∞ t =1 η 2 t < ∞ .
A typical step size schedule is η t = a ( b + t ) − r , where a, b > 0 and r ∈ (0 . 5 , 1] .
MCMC U NLEARNING A LGORITHM
This section presents the main results of this paper.
We ﬁrst deﬁne a new notion, ε -knowledge removal, to assess the machine unlearning performance in Bayesian inference.
Then, an MCMC unlearning algorithm is designed with knowledge removal and generalizability guarantees.
4. ε -K NOWLEDGE R EMOVAL FOR B AYESIAN I NFERENCE
Suppose a client requests to remove her/his data S (cid:48)
⊂ S from the whole training dataset S .
An unlearning algorithm A for Bayesian inference aims to remove the knowledge learned from the requested data S (cid:48) from the inferred distribution ˆ p S as follows,
[Equation]
where ˆ p − S (cid:48) S is named as the processed distribution .
In order to meet the regulation requirements, the following notion is deﬁned to quantify the machine unlearning performance in Bayesian inference.
Deﬁnition 1 ( ε -knowledge removal) .
For the distribution ˆ p S that inferred on dataset S via Bayesian inference and any subset S (cid:48) ⊂ S , we call algorithm A performs ε -knowledge removal, if
[Equation]
where ˆ p − S (cid:48) S =
A (ˆ p S , S (cid:48) ) is the processed distribution.
Remark 1.
Intuitively, a smaller ε indicates the algorithm A has better unlearning performance.
In practice, one usually can only obtain an inferred distribution ˆ p S ( ·| ω ) , subjects to a random seed ω .
In this case, the overall inferred distribution is ˆ p S =
(cid:82) ˆ p S ( ·| ω ) p ( ω )d ω , which suggests that the
KL-divergence in Deﬁnition 1 can then be upper-bounded as below,
[Equation]
Motivated by the above inequality, the following knowledge removal estimator is further deﬁned to efﬁciently estimate the knowledge removal performance in practice.
Deﬁnition 2 (knowledge removal estimator) .
Suppose ω 1 , · · · , ω M is M i.i.d.
random seed in
Bayesian inference.
Then, the knowledge removal estimator ˆ ε M is deﬁned as below,
[Equation]
where ˆ p − S (cid:48) S ( ·| ω m )
=
A (ˆ p S ( ·| ω m ) , S (cid:48) ) is the processed distribution for the m -th random seed ω m .
Remark 2.
Our experiments have employed this knowledge removal estimator ˆ ε M to estimate the ε value in the ε -knowledge removal guarantee.
See Section 5.2 for details.
Remark 3.
This estimator is similar to the “Local Forgetting Bound” deﬁned in Golatkar et al.
The difference is that the bound in Golatkar et al.
(2020) aims to estimate the randomness introduced by some “readout functions”, while ˆ ε M aims to quantify the ε -knowledge removal guarantee by estimating the difference between the original and processed distributions.
4. M ETHODOLOGY
This section presents the MCMC unlearning algorithm.
To make the analysis easier, we assume that one can obtain the exact targeted posterior distribution via MCMC.
In other words, for a posterior ˆ p S that is inferred by MCMC, we assume that ˆ p S = p S .
Developing unlearning algorithm for MCMC is challenging, since there is no explicitly parameterized distribution for knowledge removal analysis.
To tackle this challenge, we re-formulate the
MCMC unlearning problem as minimizing the following KL-divergence,
[Equation]
where p S is the distribution already learned via MCMC, δ is the distribution shifting scale, S (cid:48) is the dataset that is requested to be deleted, and p S − S (cid:48) is the targeted posterior that one aims to obtain.
The idea behinds Eq.
(1) is to approximate the targeted posterior p S − S (cid:48) via directly shifting the currently learned distribution p S .
In this case, one no longer needs to know the exact implicit distribution parameter to perform machine unlearning, but only the explicit shifting scale δ is enough.
So far, the MCMC unlearning problem has been converted to an optimization problem deﬁned as
Eq.
We then design MCMC unlearning algorithm based on this problem conversion.
Let δ − S (cid:48)
S denotes any local minimizer of Eq.
(1), and the goal of MCMC unlearning is to calculate the optimal shifting scale δ − S (cid:48) S .
To achieve that goal, we ﬁrst expand the KL-divergence in Eq.
(1) as follows,
[Equation]
Through Eq.
one can ﬁnd that when removing dataset S (cid:48) via optimizing Eq.
(1), an additional term E θ ∼ p S log p ( S (cid:48) | θ + δ ) is introduced to force the distribution p S ( · − δ ) approaching the target posterior p S − S (cid:48) .
This motivate us to follow existing works (Koh & Liang, 2017; Guo et al., 2020) to conduct inﬂuence analysis on Eq.
(1) based on the introduced term E θ ∼ p S log p ( S (cid:48) | θ + δ ) .
Speciﬁcally, we deﬁne a new function F − S (cid:48) ,τ (
[Equation] [Equation]
=
[Equation]
− log p ( S, θ + δ ) − τ · log p ( S (cid:48) | θ + δ )] , where τ ∈ [ − 1 , 0] .
Suppose there exists a function ˆ δ ( τ ) with domain [ − 1 , 0] such that ˆ δ (0) = 0 , and ˆ δ ( τ ) is a local minimizer of F − S (cid:48) ,τ ( δ, S ) .
Then, one can let the target shifting scale δ − S (cid:48)
S be ˆ δ ( − 1) , and approximate it by the ﬁrst-order approximation: ˆ δ ( − 1)
≈ ˆ δ (0) − ∂ ˆ δ (0) /∂τ , which means δ − S (cid:48)
S ≈ − ∂ ˆ δ (0) /∂τ .
Following a standard derivation, one can calculate the ﬁrst derivative ∂ ˆ δ (0) /∂τ
as the following deﬁned MCMC inﬂuence function I ( S (cid:48) ) .
Thus, the optimal δ − S (cid:48)
S is then approximated as δ − S (cid:48)
S ≈ −I ( S (cid:48) ) .
For the detailed inﬂuence analysis, see Appendix A.1.
Deﬁnition 3 (MCMC inﬂuence function) .
The MCMC inﬂuence function of dataset S (cid:48) ⊂ S is deﬁned to be
[Equation]
The MCMC inﬂuence function is deﬁned to estimate the inﬂuence of data S (cid:48) on the learned distribution p S , as δ − S (cid:48)
S ≈ −I ( S (cid:48) ) .
The computational cost of the MCMC inﬂuence function would be high.
An efﬁcient calculation method is given in Appendix B.
Finally, based on Deﬁnition 3, the MCMC unlearning algorithm is designed as follows.
Algorithm 1 (MCMC unlearning) .
Suppose one have drawn a series of samples { θ 1 , · · · , θ T } from the posterior p S via MCMC.
Then, MCMC unlearning algorithm removes the learned knowledge of dataset S (cid:48) from each drawn sample θ i as follows,
[Equation]
where I ( S (cid:48) ) is the MCMC inﬂuence function for dataset S (cid:48)
⊂ S .
Remark 4.
Algorithm 1 is equivalent to the following unlearning process,
[Equation]
Remark 5.
Due to the inherent geometric difference between the distribution functions p S and p S − S (cid:48) , it is impossible for Algorithm 1 to directly shift p S to exactly recover p S − S (cid:48) .
However, the experiments (see Section 5) show that the designed algorithm is good enough for removing learned knowledge from MCMC distributions.
4. K NOWLEDGE R EMOVAL A NALYSIS
This section studies the ε -knowledge removal for the proposed MCMC unlearning algorithm.
We ﬁrst introduces several assumptions (Assumptions 1-4) for the knowledge removal analysis.
Assumption 1.
Function − E θ ∼ p S log p ( θ + δ ) and − E θ ∼ p S log p ( z | θ + δ ) are L 1 -Lipschitz continuous on the support set supp( δ ) .
The Lipschitzness of − E θ ∼ p S log p ( θ + δ ) can be realized by choosing appropriate prior distribution for θ , for example, let p ( θ ) be a Laplace distribution.
Besides, the Lipschitzness of − E θ ∼ p S log p ( z | θ + δ ) can usually be achieved by setting p ( z | θ + δ ) to be a neural network with bounded domain in practice.
Assumption 2.
There exists a neighborhood V 2 ⊂ supp( δ ) of the origin point such that for any dataset S and any sample point z ∈ Z , the followings hold:
• (Smoothness).
∇ δ [ − E θ ∼ p S log p ( θ + δ )] and ∇ δ [ − E θ ∼ p S log p ( z | θ + δ )] are L 2 -Lipschitz continuous on the space V 2 .
• (Lipschitz Hessian).
∇ 2 δ [ − E θ ∼ p S log p ( θ + δ )] and ∇ 2 δ [ − E θ ∼ p S log p ( z | θ + δ )] are L 3Lipschitz continuous on the space V 2 .
Assumption 2 assumes the local smoothness and Hessian Lipschitzness for the involved functions, which are common in literature for analyzing Newton methods (Nesterov & Polyak, 2006; Carmon et al., 2018;
Zhou et al., 2018).
Assumption 3 (strong convexity) .
There exists a neighborhood V 3 ⊂ supp( δ ) of the origin point such that for any dataset S and any sample point z ∈ Z , − E θ ∼ p S log p ( θ + δ ) and − E θ ∼ p S log p ( z | θ + δ ) are µ -strongly convex on V 3 .
The inﬂuence analysis and the deﬁned MCMC inﬂuence function require the Hessian matrices of − E θ ∼ p S log p ( θ + δ ) and − E θ ∼ p S log p ( z | θ + δ ) to be invertible.
Assumption 3 ensures the invertibilities of these Hessian matrices.
Assumption 4.
Suppose Assumptions 1-3 hold.
Then there exists a sub-neighborhood V ∈
V 2 ∩ V 3 of the origin point such that for any dataset S and any real τ ∈ [ − 1 , 0] , the function − E θ ∼ p S log p ( θ + δ | S ) − τ · E θ ∼ p S log p ( S (cid:48) | θ + δ ) has at least one local minimizer falls in V .
Assumption 4 assumes the target optimal solution δ − S (cid:48) S of Eq.
(1) exists in the subspace that we are interested in, which would make the inﬂuence analysis being meaningful.
Under Assumptions 1-4, we prove that the MCMC inﬂuence function deﬁned in Deﬁnition 3 can rigorously estimate the inﬂuence of data on the shifting scale δ .
Theorem 1.
Under Assumptions 1-4, we have that
[Equation]
The proof of Theorem 1 is presented in Appendix A.1.
Based on Theorem 1, a knowledge removal guarantee is proved for the MCMC unlearning algorithm.
Theorem 2.
Suppose Assumptions 1-4 hold.
Then, the MCMC unlearning algorithm A ( p S , S (cid:48) )
= p S ( · + I ( S (cid:48) )) performs ε − S (cid:48)
S -knowledge removal, where
[Equation]
and δ − S (cid:48)
S is a local minimizer of the KL-divergence KL( p S ( · − δ ) (cid:107) p S − S (cid:48) ) .
Proof sketch.
We expand KL( p S ( · + I ( S (cid:48) ))
(cid:107) p S − S (cid:48) ) into Taylor series up to 2 -th order around the point δ = δ − S (cid:48) S .
By applying the ﬁrst-order optimality condition, the ﬁrst-order term in the Taylor series becomes 0 .
By leveraging the approximation error given in Theorem 1, the second-order term is proved to be no larger than the order of O ( | S (cid:48) | 4 /N 3 ) .
The proof is presented in Appendix A.2.
Remark 6.
As explained in the previous section, Algorithm 1 could not completely remove the learned knowledge from the inferred distribution due to the inherent geometric difference between the distribution p S and p S − S (cid:48) .
Nevertheless, Theorem 2 demonstrates that the Algorithm 1 can help the KL-divergence in Eq.
(1) approach its local minimum.
4. G ENERALIZATION A NALYSIS
This section studies the impact of the proposed MCMC unlearning algorithm on the generalization ability of the inferred distribution.
The analysis is conducted based on the PAC-Bayes theory (McAllester, 1999; 2003; He et al., 2019), a framework for analyzing the generalization ability of stochastic machine learning models (Mohri et al., 2012; He & Tao, 2020).
            2ULJLQDO'DWD             6*/' 2ULJLQ7DUJHW3URFHVVHG             6*+0& 2ULJLQ7DUJHW3URFHVVHG
6*/' 6*+0&     
   8 Q O HD U Q L QJ  7 L P H   V  7LPH&RVW 5HWUDLQ2XUV
[Figure]
Speciﬁcally, suppose Q is the parameter distribution of a stochastic model.
Suppose S is the training dataset.
Then, the expected risk R ( Q ) and empirical risk ˆ R S ( Q ) of Q are deﬁned to be
[Equation]
where θ is the model parameter drawn from the distribution Q and (cid:96) is a loss function ranging in [0 , 1] .
The difference of the expected risk and the empirical risk is the generalization error.
Its magnitude characterizes the generalization ability of the stochastic model.
Then, a generalization bound for the distribution processed by Algorithm 1 is proved as below.
Theorem 3.
Suppose Assumptions 1-4 hold.
Let p − S (cid:48)
S denotes the distribution processed by Algorithm 1.
Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ , the following inequality holds,
R ( p − S (cid:48) S ) ≤ ˆ R S
( p − S (cid:48) S )
+ (cid:115) E θ
[Equation]
p S [log p S
[Equation]
θ ) + (cid:107)
[Equation]
(cid:107)
[Equation]
+ (cid:107)I ( S
[Equation]
) (cid:107)
[Equation]
+ log
[Equation]
+ log
[Equation]
+ log N +2 N −, where (cid:107)I ( S (cid:48) ) (cid:107) 1 ≤ O ( | S (cid:48) | /N ) .
The proof is given in Appendix A.3.
Corollary 1.
Algorithm 1 increases the generalization upper bound in Theorem 3 no larger than the order of O ( (cid:112) | S (cid:48) | /N ) .
Proof.
According to Theorem 3, the difference of the generalization upper bound introduced by
Algorithm 1 is O ( (cid:113) (cid:107)I ( S (cid:48) )
(cid:107) 1 2 N − 1 ) ≤ O ( (cid:113) O
( | S (cid:48) | /N )
2 N − 1 )
=
O ( √ | S (cid:48)
|
N ) .
This completes the proof.
Remark 7.
Corollary 1 indicates that the proposed MCMC unlearning algorithm would not compromise the generalization ability of the inferred distribution.
E XPERIMENTS
In this section, we empirically verify the effectiveness and efﬁciency of the proposed MCMC unlearning algorithm on the Gaussian mixture models and Bayesian neural networks.
5. E XPERIMENTS FOR G AUSSIAN M IXTURE M ODELS
We ﬁrst visualize the effect of the MCMC unlearning algorithm on a synthetic clustering dataset that consists of 2 , 000 examples, where each examples is two-dimensional and is possibly fromclusters.
The Gaussian mixture models (GMMs) are employed to infer the cluster centers.
For the details of the GMMs used in our experiments, see Appendix C.1.
Experiment design.
We employ SGLD and SGHMC to infer the GMMs on the clustering dataset.
Then, 400 points are removed from each of the grey parts and yellow parts, around 40% of the whole
[Table]
dataset at all, by our forgetting algorithms.
We also trained models on only the remaining set with the same MCMC inference settings to show the targets of the forgetting tasks.
For the details of the experiments, see Appendix C.
Results analysis.
The visualization results are presented in Fig.
1a, which show that after unlearning, the processed models are close to the target models.
Besides, the time costs for processing single data deletion requests are presented in Fig.
1b, which shows that the proposed MCMC unlearning algorithm is signiﬁcantly faster than retraining the whole model.
These results demonstrates that the proposed algorithm can effectively and efﬁciently remove the learned knowledge of speciﬁed data.
5. E XPERIMENTS FOR B AYESIAN N EURAL N ETWORKS
6*/' 6*+0&    8 Q O HD U Q L QJ  7 L P H
  V
&,)$5 5HWUDLQ,62XUV
We then apply the MCMC unlearning algorithm to the
Bayesian neural networks on the CIFAR-10 (Krizhevsky et al., 2009) dataset for classiﬁcation.
Analogous experiments are also conducted on the Fashion-MNIST (Xiao et al., 2017) dataset, please see Appendix D for more details.
Dataset & BNN.
We divide the training set S of CIFARinto two parts, the removed part S f and the remained part S r .
A certain number of examples are randomly chosen from a single class in the dataset to form the removed training set S f .
The test set is denoted by S test .
A BNN consists of two convolutional layers and two fully-connected layers is adopted in the experiments.
For more details about the dataset and model architecture, see Appendix D.
Baseline method.
We compare the proposed MCMC unlearning algorithm with the importance sampling method (IS).
Speciﬁcally, when a data deletion request comes, the importance sampling method will process the request via performing MCMC sampling on the remaining data.
Evaluation metrics.
Four kinds of metrics are used to evaluate the performance of machine unlearning: (1) Classiﬁcation errors.
We calculate classiﬁcation errors the remaining set S r , removed set S f , and test set S test for different models.
Ideally, after removing data, the three errors of the processed model would approach that of the retrained model.
(2) ε -Knowledge removal guarantee.
We use the knowledge removal estimator ˆ ε M deﬁned in Deﬁnition 2 to estimate the ε value in the ε -knowledge removal guarantee.
A smaller estimation result would imply a better knowl-
S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H
5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U
X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV
S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H
5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U
X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV
S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H
5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U
X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV
S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H
5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U
X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV S O DQH F D U E L U G F D W GHH U GRJ I U RJKR U V H V K L S W U X FN 
     & RQ I L GHQ F H 5HPRYHGVDPSOHV
[Figure]
edge removal performance.
Calculating ˆ ε M requires to ﬁx a series of random seeds ω 1 , · · · , ω M .
To this end, we use a series of pre-trained models as the random seeds.
See Appendix D.3 for the detailed calculation procedures.
(3) Membership inference attack (MIA) accuracy.
We employ a white-box MIA (Yeom et al., 2018) to infer whether the examples from the removed subset S f come from the training set of the model.
Intuitively, the attack accuracy would decline after unlearning.
Thereby, a small attack accuracy implies a strong knowledge removal performance.
See Appendix
D.4 for more details about the MIA and its calculation procedures.
(4) Time costs for unlearning.
For different unlearning methods, we estimate the average time usages for processing single data deletion requests.
A smaller time cost corresponds to high unlearning efﬁciency.
Experiment designs.
We ﬁrst train the BNN on the complete training set S with SGLD and
SGHMC, respectively.
Then, the subset S f is removed iteratively, where a ﬁxed number of data points are removed in each iteration.
We also trained models on only the remaining set S r to show the targets of the machine unlearning task.
For the details of the experiments, see Appendix D.5.
Results analysis.
The experiment results are presented in Table 1.
In all experiments, we observe that the proposed MCMC unlearning algorithm can signiﬁcantly increase the model error on sample set S f while making the error on sample sets S r and S test close to that of the retrained one.
Besides, our algorithm can also reduce the ε value in the knowledge removal guarantee and the MIA accuracy on S f , which further indicates it can indeed remove speciﬁed knowledge from the MCMC models.
In contrast, the importance sampling method could neither effectively make the classiﬁcation errors approach the targets, nor achieve stronger ε -knowledge removal.
We also present the time costs for different unlearning methods in Fig.
One can found that the MCMC unlearning algorithm is signiﬁcantly faster than other methods.
All these experiment results demonstrate the effectiveness and efﬁciency of the proposed algorithm.
Finally, we give a case study about the prediction changes on the test set examples to illustrate the effect of the proposed MCMC unlearning algorithm.
The results are presented in Fig.
3, where one can observe that as the data removing continues, the prediction conﬁdence of the model on the removed class signiﬁcantly decreases, while those on other classes remain the same or increase.
C ONCLUSION
The right to be forgotten imposes a considerable compliance burden on AI companies.
A company may need to delete the whole model learned from massive resources due to a request to delete a single sample point.
Existing works only appliable to explicit parameterized optimization problem, which however not work for sampling-based Bayesian inference method, i.e. , MCMC.
In this work, we convert the MCMC unlearning problem to an optimization problem.
Then, an MCMC inﬂuence function is designed to characterize the inﬂuence of data on the distribution inferred by MCMC.
It then delivers the ﬁrst MCMC unlearning algorithm.
Theoretical analyses show that the proposed algorithm can indeed reduce the learned knowledge, and would not compromise the generalization ability of the inferred distribution.
Experiments show that the proposed algorithm can remove the inﬂuence of speciﬁed samples without compromising the knowledge learned on the remained data.
A CKNOWLEDGMENTS
FH is supported in part by the Major Science and Technology Innovation 2030 “New Generation
Artiﬁcial Intelligence” key project (No. 2021ZD0111700).
SF and DT are supported by Australian
Research Council Projects FL-170100117, IH-180100002, IC-190100031, and LE-200100049.
The authors sincerely appreciate Yue Xu and the anonymous ICLR reviewers for their helpful comments
A P ROOFS
This section collects all the proofs in this paper.
To avoid technicalities, we assume that all functions are differentiable throughout this paper.
Furthermore, the order of differentiation and integration is assumed to be interchangeable.
For simplicity, we denote that
• h
( δ, z ) :
= − E θ ∼ p S log p ( z | θ + δ ) , • f ( δ ) :
= − E θ ∼ p S log p ( θ + δ ) , • F ( δ, S ) :
= (cid:80) z ∈ S h
( δ, z ) + f ( δ ) =
(cid:80) z ∈ S − E θ ∼ p S log p ( z | θ + δ ) − E θ ∼ p S log p ( θ + δ ) .
A. P ROOF OF T HEOREM
To prove Theorem 1, we ﬁrst deﬁne the following function,
[Equation]
where τ ∈ [ − 1 , 0] . Under Assumption 3, it can be shown that F − S (cid:48) ,τ ( δ, S ) is strongly convex on the space V 3 deﬁned in Assumption 3.
Lemma 1.
Suppose Assumption 3 holds.
Then, for any τ ∈ [ − 1 , 0] , F − S (cid:48) ,τ ( δ, S ) is strongly convex on the space V 3 .
Proof of Lemma 1.
We rearrange the function F − S (cid:48) ,τ as follows,
[Equation]
Apparently, F ( δ, S − S (cid:48) ) is strongly convex on V 3 .
Since τ ∈ [ − 1 , 0] , we have that 1+ τ ≥ 0 .
Thus, (
[Equation]
+ τ )
[Equation]
z ) is either strongly convex on V 3 or equal to zero.
Therefore, F − S (cid:48) ,τ ( δ, S ) is strongly convex on V 3 .
The proof is completed.
We then prove that when Assumption 4 holds, there exists a continuous mapping that “connects” the origin point 0 ∈ R d and the target shifting scale δ − S (cid:48) S .
Lemma 2.
Suppose Assumption 4 holds.
Then, there exists a continuous function ˆ δ : [ − 1 , 0]
→ V such that for any τ ∈ [ − 1 , 0] , ˆ δ ( τ ) is the global minimizer of the function F − S (cid:48) ,τ ( δ, S ) on space V .
Proof of Lemma 2.
By applying Lemma 1 and Assumption 4, we have that the global minimizer of the function F − S (cid:48) ,τ ( γ, S ) uniquely exists in V .
Therefore, one can deﬁne the mapping ˆ δ as below,
[Equation]
where τ ∈ [ − 1 , 0] .
We then prove the continuity of ˆ δ .
Notice that ˆ δ ( τ ) is the solution of the following equation,
[Equation]
Since F − S (cid:48) ,τ ( δ, S ) is strongly convex, thus its Hessian matrix ∇ 2 δ F − S (cid:48) ,τ (ˆ δ ( τ ) , S ) is invertible.
Combining with the implicit function theorem, we have that ˆ
δ is continuous on the interval [ − 1 , 0] .
The proof is completed.
When τ takes 0 and − 1 , the function ˆ δ ( τ ) becomes 0 ∈ R d and δ − S (cid:48) S , respectively.
Therefore, δ − S
[Equation]
can be expanded into Taylor series as follows,
[Equation]
where ξ ∈ [ − 1 , 0] , 12 ∂ 2 ˆ δ ( ξ ) ∂τ 2 is the Lagrange form of the remainder.
We prove that the Lagrange remainder term becomes negligible as the training set size N goes to inﬁnity.
To do this, we ﬁrst prove the following Lemma.
Lemma 3.
Suppose Assumptions 1-4 hold.
The mapping ˆ δ is as deﬁned in Lemma 2.
Then, we have that
[Equation]
and for any τ ∈ [ − 1 , 0] , (cid:107) ∂ ˆ δ ( τ ) ∂τ (cid:107) 2 ≤ O ( | S (cid:48) | /N ) .
Proof.
We ﬁrst calculate ∂ ˆ δ ( τ ) ∂τ based on Eq.
(3) in the proof of Lemma 2.
Calculate the derivatives of the both sides of Eq.
(3) with respect to τ , we have that
[Equation]
According to Lemma 1, F − S (cid:48) ,τ ( δ, S ) is strongly convex on V .
Thus, the following Hessian matrix
[Equation]
is positive deﬁnite, hence invertible.
Combining with Eq.
(5), we have that
[Equation]
We then upper bound the norm of ∂ ˆ δ ( τ ) ∂τ .
Based on Eq.
(6), we have that
[Equation]
We ﬁrst consider the ﬁrst term of the right-hand side of Eq.
By Assumption 3, both f ( δ ) and h ( δ, z ) are µ -strongly convex on the space V ⊂ supp( δ ) .
Thus, we have that
[Equation]
Let λ min denotes the smallest eigenvalue of the following matrix
[Equation]
Then, the Eq.
(8) implies that λ min ≥ N −| S (cid:48) | N µ .
Hence, we have the following,
[Equation]
We then upper bound the second term of the right-hand side of Eq.
By applying Assumption 1, we have that
[Equation]
Finally, inserting eqs.
(9) (10) into Eq.
[Equation]
we eventually have that
[Equation]
The proof is completed.
Then, the following Lemma is obtained based on Lemma 3, which demonstrates the strictness of the approximation in Eq.
Lemma 4.
Suppose Assumptions 1-4 hold.
The induced mapping ˆ δ is as deﬁned in Lemma 2.
Then, for any τ ∈ [ − 1 , 0] , we have that (cid:107) ∂ 2 ˆ δ ( τ ) ∂τ 2 (cid:107) 2 ≤ O ( | S (cid:48) | 2 /N 2 ) .
Proof.
We ﬁrst calculate ∂ 2 δ ( τ ) ∂τ 2 based on Eq.
Similar to the proof of Lemma 3, we calculate the second-order derivatives of the both sides of Eq.
(3) with respect to τ and have that
[Equation]
which means
[Equation]
in which the invertibility of (cid:16) 1 N (cid:80)
[Equation]
[Equation]
ˆ
[Equation]
, S ) + τN
[Equation]
z ∈ S
[Equation]
∇ 2 δ h (ˆ δ ( τ ) , z ) (cid:17) is guaranteed by Eq.
(8), A ( τ ) , B
( τ, z ) ∈ R K × 1 , and for i = 1 , · · · , K , we have the following,
[Equation]
We then upper bound the norm of ∂ 2 ˆ δ ( τ ) ∂τ 2 .
Based on Eq.
(11), we have that
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation]
[Table]
[Equation]
where Eq.
(15) is obtained by inserting Eq.
(9) (in the proof of Lemma 3) into Eq.
Thus, the remaining task is to upper bound the norms of A ( τ ) , B ( τ, z ) and ∇ 2 δ (cid:80) z ∈ S
[Equation]
h (ˆ δ ( τ ) , z ) · ∂ ˆ δ ( τ ) ∂τ .
We ﬁrst upper bound (cid:107) A ( τ ) (cid:107) 2 .
Applying Lemma 3, we have that (cid:107) ∂ ˆ δ ( τ ) ∂τ (cid:107) 2 ≤ O ( | S (cid:48) | /N ) .
Applying the Lipschitz Hessian condition in Assumption 2, we have that (cid:107)∇ 2 δ ( ∂f (ˆ δ ( τ )) ∂δ i (cid:107) 2 is also bounded by a real constant.
Therefore, we have that
[Equation]
For B ( τ, z ) , we similarly have that
[Equation]
[Equation]
To upper bound the norm of (cid:80) z ∈ S (cid:48) ∇ 2 δ h (ˆ δ ( τ ) , z ) · ∂ ˆ δ ( τ ) ∂τ , we apply Lemma 3, Assumption 2, and have that
[Equation]
Inserting eqs.
(16), (17) and (18), into Eq.
(15), we eventually have that
[Equation]
The proof is completed.
Finally, combining all the results, we prove Theorem 1.
Proof of Theorem 1.
The proof is completed by applying Lemmas 3 and 4 into Eq.
A. P ROOF OF T HEOREM
This section presents the proof of Theorem 2.
Proof of Theorem 2.
By expanding the KL-divergence KL( p S ( · + I ( S (cid:48) )) (cid:107) p S − S (cid:48) ) into Taylor series around the local region of δ = δ − S (cid:48) S , we have
[Equation]
where the ﬁrst-order term equals 0 according to the ﬁrst-order optimal condition
[Equation]
By applying
Theorem 1 and Assumption
[Equation]
we further have
[Equation]
[Equation]
By combining eqs.
(19) and (20), the proof is completed.
A. P ROOF OF T HEOREM
This section presents the proof of Theorem 3.
We derive generalization bound for the proposed algorithm under the PAC-Bayesian framework (McAllester, 1998; 1999; 2003).
The framework can provide guarantees for randomized predictors ( e.g. , the Bayesian predictors).
Speciﬁcally, let Q a distribution on the parameter space Θ , P denotes the prior distribution over the parameter space Θ . Then, the expected risks R ( Q ) is bounded in terms of the empirical risk ˆ R ( Q, S ) and KL-divergence KL( Q (cid:107) P ) by the following result from PAC-Bayes.
Lemma 5 (cf.
McAllester (2003), Theorem 1) .
For any real δ ∈ (0 , 1) , with probability at least 1 − δ , we have the following inequality for all distributions Q :
[Equation]
Based on Lemma 5, we prove the generalization bounds in Theorem 3.
proof of Theorem 3.
Let the prior distribution P be a Laplace distribution lap(0 , 1) ,
p − S (cid:48) S = p S ( · +
I ( S (cid:48) )) be the distribution processed by the MCMC unlearning algorithm.
Then, one can calculate the KL-divergence KL( p − S (cid:48) S (cid:107) P ) as follows (where we assume that Θ = R d ),
[Equation]
Inserting Eq.
(22) into Eq.
(21) in Lemma 5, we then obtain the PAC-Bayesian generalization bound.
Eventually, by applying Lemma 3, we have that (cid:107)I ( S (cid:48) ) (cid:107) 1 ≤ O ( | S (cid:48) | /N ) .
The proof is completed.
B E FFICIENT C ALCULATION OF MCMC I NFLUENCE F UNCTION
A major computing burden in the MCMC unlearning algorithm is calculating the product of H − 1 v , where H is the Hessian matrix of some vector-valued function f ( x ) and v is a constant vector.
When the function f ( x ) is of high dimension, the calculation would have a considerably high computational cost.
We follow Agarwal et al.
(2017) and Koh & Liang (2017) to apply a divide-and-conquer strategy to address the issue.
This strategy relies on calculating the Hessian-vector product Hv .
Hessian-vector product (HVP).
We ﬁrst discuss how to efﬁciently calculate Hv .
The calculation of Hv can be decomposed into two steps: (1) calculate ∂f ( x ) ∂x and then (2) calculate ∂∂x (cid:16) ∂f ( x ) ∂x · v (cid:17) .
It is worth noting that ∂f ( x ) ∂x ∈ R 1 × d and v ∈ R d × 1 , where d > 0 is the dimension of data.
Thus, (cid:16) ∂f ( x ) ∂x · v (cid:17) is a scalar value.
Calculating its gradient ∂∂x (cid:16) ∂f ( x ) ∂x · v (cid:17) has a very low computational cost on platform PyTorch (Paszke et al., 2017) or TensorFlow (Abadi et al., 2015).
Calculating H − 1 v .
When the norm (cid:107) H (cid:107) ≤ 1 , the matrix H − 1 can be expanded by the Taylor’s series as H − 1 = (cid:80) ∞ i =
( I − H ) i .
Deﬁne that H − 1 j = (cid:80) ji =
( I − H )
i .
Then, we have the following recursive equation,
[Equation]
Agarwal et al.
(2017) prove that when j → ∞ , we have E
[ H − 1 j ] → H − 1 .
Therefore, we employ
H − 1 j v to approximate H − 1 v .
Moreover, to secure the condition (cid:107) H (cid:107) ≤ 1 stands, we scale H to cH by a scale c ∈ R + , such that (cid:107) cH (cid:107) ≤ 1 .
Then, we approximate ( cH ) − 1 .
Eventually, we have that H − 1 =
c ( cH ) − 1 .
We can plug it to the applicable equations above.
C E XPERIMENTS D ETAILS FOR G AUSSIAN M IXTURE M ODELS
This section provides the additional experiments details for the Gaussian mixture models.
C. G AUSSIAN M IXTURE M ODELS
We employ Gaussian mixture models (GMMs) to infer the cluster centers on the synthetic dataset.
A
GMM assumes that each example is drawn from K Gaussian distributions centered at µ 1 , · · · , µ K , respectively.
Then, the hierarchical structure of GMM is as follows: (1) draw a clustering center from the uniform distribution over { µ c 1 , . .
. , µ c K } ; and (2) sample
z i from a Gaussian distribution centering at µ c i .
That is,
[Equation]
where 1 ≤ k ≤ K , 1 ≤ i ≤ n , µ k ∈ R d , c i ∈ { 1 , · · · , K } , Z i ∈ R d , and the hyperparameter σ ∈ R is the prior standard deviation.
In our experiments, the factor K is set as 4 , and the prior standard deviation σ is set as 1 .
C.
E XPERIMENTS D ETAILS
The experiments for GMM have two main phases:
Training phase.
Every GMM is trained for 4 , 000 iterations.
The batch size is set as 64 .
For both SGLD and SGHMC, the learning rate schedule is set as 4 · t − 0 .
5005 /N , where t is the training iteration step and N is the number of the training set size.
Besides, the momentum factor α of
SGHMC is set as 0 .
Unlearning phase.
We remove a batch of 4 datums each time.
When calculating the inversedHessian-vector product H − 1 v in the inﬂuence functions (see Section B), the recursive calculation number j is set as 32 , and the scaling factor c is set as 1 /N (cid:48) , where N (cid:48) is the number of the current remained training examples.
Notice that N (cid:48) will gradually decrease as the forgetting process continues.
Moreover, we employ the Monte Carlo method to calculate the expectations in the MCMC inﬂuence function.
Speciﬁcally, we repeatedly draw sample θ for 5 times, calculate the matrix or vector in the MCMC inﬂuence function, and average the results to approach the expectations.
D E XPERIMENTS D ETAILS FOR B AYESIAN N EURAL N ETWORKS
This section provides the additional experiments details for Bayesian neural networks.
D. D ATASETS
We employ two image datasets, Fashion-MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al., 2009), in our experiments.
Fashion-MNIST consists of grey-scale images from 10 classes, where each class contains 6 , 000 training examples and 1 , 000 test examples.
Besides, CIFARconsists of color images from 10 classes, where each class contains 5 , 000 training examples and 1 , 000 test examples.
No data augmentation is used in the experiments.
Furthermore, it is worth noting that the models used for Fashion-MNIST and CIFAR-10 are ﬁrst pretrained on MNIST (LeCun et al., 1998) and CIFAR-100 (Krizhevsky et al., 2009), respectively.
Please see Appendices D.3 and D.5 for more details.
D. B AYESIAN N EURAL N ETWORKS
This section introduces the model architectures used in our experiments and the implementation details for training BNNs.
D.2. M ODEL A RCHITECTURES
For Fashion-MNIST, we use a LeNet (LeCun et al., 1998) architecture that consists of two convolutional layers followed by three fully-connected layers.
The convolutional layers use 5 × 5 convolutions, each followed by a 2 × 2 max-pooling layer and a ReLU activation function.
The channel numbers of the two convolutional layers are 6 and 16 , respectively.
Besides, each fully-connected layer is also followed by a ReLU activation function, while the hidden layer channels are 120 and 84 , respectively.
for CIFAR-10, we follow Abadi et al.
(2016) to use a network architecture that consists of two convolutional layers followed by two fully-connected layers.
Each convolutional layer uses 5 ×convolution, with a channel number of 64 , followed by a 2 × 2 max-pooling layer and a ReLU activation function.
Besides, each fully-connected layer is also followed by a ReLU activation function, while the hidden layer channels are 384 .
D.2. M ODEL T RAINING
In all the experiments, we use an isotropic Gaussian distribution N (0 , σ 20 I ) as the prior of the BNNs, in which the hyperparameter σ 0 is set as 0 . 1 .
Traditional SG-MCMC methods would inject random noise to BNNs during training.
However, it is found that the noise injection in the early training iterations hurts the convergence of BNNs (Zhang et al., 2020a).
To alleviate such a problem, we follow Zhang et al.
(2020a) to ﬁrst avoid the noise injection in the early training iterations and then resume SG-MCMC as usual in the rest of the training.
D. E STIMATING THE ε -K NOWLEDGE R EMOVAL G UARANTEE
We use the knowledge removal estimator ˆ ε M deﬁned in Deﬁnition 2 to estimate the ε value in the εknowledge removal guarantee.
According to the deﬁnition of ˆ ε M , calculating it requires ﬁrst ﬁxing a series of random seeds, and then calculating the KL-divergence between two MCMC distributions.
For the random seeds, we employ the pretraining technique (Golatkar et al., 2020) to ﬁx them.
Speciﬁcally, for the experiments on Fashion-MNIST, all the models used are ﬁrst pre-trained on
MNIST; for the experiments on CIFAR-10, all the models used are ﬁrst pre-trained on CIFAR-100.
Besides, for the KL-divergence calculation, we follow Wang et al.
(2009) to estimate the KLdivergence in a k -NN manner, in which 100 samples are drawn from each of the processed and retrained distributions for the estimation.
D. M EMBERSHIP I NFERENCE A TTACK
Membership inference attack (MIA) (Shokri et al., 2017; Yeom et al., 2018) is a privacy attack that aims to infer whether a given example is in the training set based on the output of the model.
In our experiments, we adopt a white-box threshold-based MIA (Yeom et al., 2018) to assess the unlearning performance.
The calculation consists of two steps: (1) learn an optimal MIA threshold on the remained training set S r and the test set S test , and (2) calculate MIA accuracy with the learned threshold on the removed training set S f .
Learn the optimal threshold.
Suppose the “to-be-inferred” example ( x, y ) comes from S r and
S test with equal probabilities.
Then, the attack accuracy of MIA with a threshold ρ ∈ [0 , 1] on the model f θ is calculated as follows,
[Equation]
where f θ ( x ) y is the output conﬁdence for label y and 1 [ · ] is the indicator function.
Then, the optimal threshold ρ optim is obtained via calculating the maximizer of the above attack accuracy, i.e. ,
[Equation]
Calculate MIA accuracy on S f .
With the optimal threshold ρ optim that learned on S r and S test , the MIA accuracy on the removed set S f is calculated as follows,
[Equation]
Intuitively, after removing the knowledge learned from the removed subset S f , the MIA accuracy
Acc( ρ optim , S f ) would decline.
As a result, a low MIA accuracy on the removed subset S f would imply a strong knowledge removal performance.
D. E XPERIMENTS D ETAILS
This section provides the omitted experiment details, including the settings of the hyperparameters and the procedures of the experiments.
D.5. E XPERIMENTS ON F ASHION -MNIST
Pretraining phase.
In every experiment, we pretrain a non-Bayesian model on MNIST via SGD for 2 , 000 iterations, with a batch size of 128 , a learning rate of 0 . 1 /N , where N is the training set size, and a momentum factor of 0 . 9 .
Training phase.
Every BNN is trained for 10 , 000 iterations.
The batch size is set as 128 .
For both
SGLD and SGHMC, we ﬁrst train the model without noise injection in the ﬁrst 1 , 000 iterations.
In this stage, the learning rate is ﬁxed to 0 .
01 /N .
Then, we resume the traditional SGLD and SGHMC in the rest of the training.
In this stage, the learning rate schedule is set as 0 .
01 · t − 0 .
/N , where t is the training iteration step.
Besides, the momentum factor α of SGHMC is set as 0 . 9 .
Unlearning phase.
We remove a batch of 64 datums each time.
For the MCMC unlearning algorithm, when calculating the inversed-Hessian-vector product H − 1 v in the MCMC inﬂuence function (see Appendix B), the recursive calculation number j is set as 64 , the scaling factor c is set as 0 . 05 /N (cid:48) , in which N (cid:48) is the number of the current remained training examples.
Notice that N (cid:48) will gradually decrease as the forgetting process continues.
Besides, we employ the Monte Carlo method to calculate the expectations in the MCMC inﬂuence function.
Speciﬁcally, each time we draw a sample θ via the MCMC sampler and calculate the matrix and vector in the MCMC inﬂuence function based on θ to estimate the desired expectation.
Besides, for the importance sampling method, we process each data deletion request by performing
MCMC sampling for 1 , 000 times on the remaining data.
D.5. E XPERIMENTS ON CIFAR-
Pretraining phase.
In every experiment, we pretrain a non-Bayesian model on CIFAR-100 via SGD for 10 , 000 iterations, with a batch size of 128 , a learning rate of 0 . 1 /N , where N is the training set size, and a momentum factor of 0 . 9 .
Training phase.
Every BNN is trained for 20 , 000 iterations.
The batch size is set as 128 .
For both
SGLD and SGHMC, we ﬁrst train the model without noise injection in the ﬁrst 5 , 000 iterations.
In this stage, the learning rate is ﬁxed to 0 .
01 /N .
Then, we resume the traditional SGLD and SGHMC in the rest of the training.
In this stage, the learning rate schedule is set as 0 .
01 · t − 0 .
/N , where t is the training iteration step.
Besides, the momentum factor α of SGHMC is set as 0 . 9 .
Unlearning phase.
We remove a batch of 64 datums each time.
For the MCMC unlearning algorithm, When calculating the inversed-Hessian-vector product H − 1 v in the MCMC inﬂuence functions (see Appendix B), the recursive calculation number j is set as 64 , the scaling factors c is set as 0 .
08 /N (cid:48) , where N (cid:48) is the number of the current remained training examples.
Notice that N (cid:48) will gradually decrease as the forgetting process continues.
Besides, we employ the Monte Carlo method to calculate the expectations in the MCMC inﬂuence function.
Speciﬁcally, each time we draw a sample θ via the MCMC sampler and calculate the matrix and vector in the MCMC inﬂuence function based on θ to estimate the desired expectation.
Besides, for the importance sampling method, we process each data deletion request by performing
MCMC sampling for 1 , 000 times on the remaining data.
D. E XPERIMENT R ESULTS ON F ASHION -MNIST
This section presents all the experiment results of BNNs on the Fashion-MNIST dataset.
The evaluations of unlearning performance are shown as Table 2, while the time costs for unlearning is shown in Fig.
The results further justify the effectiveness of the proposed algorithm.
[Figure]
D. E VALUATION ON P REDICTION D IFFERENCES
In this section, we adopt the metric named prediction differences to better quantify the difference between the retrained and processed models.
For two BNNs p 1 ( · ) and p 2 ( · ) learned by MCMC, their prediction difference on a given dataset S is deﬁned as E θ 1 ∼ p 1 E θ 2 ∼ p 2 1 | S | (cid:80) ( x i ,y i ) ∈ S (cid:107) f θ 1 ( x i ) −
[Table]
[Table]
f θ 2 ( x i ) (cid:107) 1 , where f θ 1 ( x i ) and f θ 2 ( x i ) are two prediction conﬁdence vectors for the example x i .
Intuitively, a smaller prediction difference indicates a smaller deviation between the retrained and processed models.
We calculate the prediction difference between the retrained model and the processed model on three different datasets, S r , S f , and S test .
The experiment results on CIFAR-10 and Fashion-MNIST are shown in Table 3.
The results show that the proposed MCMC unlearning algorithm can effectively reduce the prediction difference, which further justify the effectiveness of the proposed method.