Prompt Certiﬁed Machine Unlearning with Randomized Gradient Smoothing and Quantization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The right to be forgotten calls for efﬁcient machine unlearning techniques thatmake trained machine learning models forget a cohort of data.
The combinationof training and unlearning operations in traditional machine unlearning methodsoften leads to the expensive computational cost on large-scale data.
This paperpresents a Prompt Certiﬁed Machine Unlearning algorithm, PCMU, which executesone-time operation of simultaneous training and unlearning in advance for a seriesof machine unlearning requests, without the knowledge of the removed/forgottendata.
First, we establish a connection between randomized smoothing for certiﬁedrobustness on classiﬁcation and randomized smoothing for certiﬁed machine unlearning on gradient quantization.
Second, we propose a prompt certiﬁed machineunlearning model based on randomized data smoothing and gradient quantization.
We theoretically derive the certiﬁed radius R regarding the data change before andafter data removals and the certiﬁed budget of data removals about R .
Last but notleast, we present another practical framework of randomized gradient smoothingand quantization, due to the dilemma of producing high conﬁdence certiﬁcates inthe ﬁrst framework.
We theoretically demonstrate the certiﬁed radius R (cid:48) regardingthe gradient change, the correlation between two types of certiﬁed radii, and thecertiﬁed budget of data removals about R (cid:48) .
Introduction
In order to to respect the privacy, machine unlearning techniques aim to enable data owners toproactively remove their data and eliminate its inﬂuence from already trained machine learning modelupon requests [9, 44, 107, 43, 50, 40, 51, 128, 95].
A straightforward solution is to retrain new modelsfrom scratch on the remaining/remembered data, without the knowledge of the removed/forgottendata, as if the retrained model has never seen the forgotten data.
However, the naive method isimpractical since it often encounters expensive cost over complex models (e.g., DNN) on large data.
This has motivated the recent study of resolving the inefﬁciency issue of the naive machine unlearning.
Existing techniques can be broadly classiﬁed into two categories: (1) Exact unlearning algorithmsaim to learn an unlearning model with the same performance as the above naive ones retrainingfrom scratch by completely excluding the forgotten data from the training data [103, 11, 61, 16,77, 83, 7, 104, 43, 9, 3, 6, 118, 14, 13] and (2) Approximate unlearning methods try to bring theparameters of the trained model closer to the naive ones through the relaxation of exact unlearningrequirements [3, 49, 46, 116, 78, 85, 129, 57, 92, 63, 43, 50, 92, 106, 44, 45, 50, 94, 47, 38, 78, 79,81, 120, 127, 46, 14, 136, 80].
Certiﬁed removal is a certiﬁed-removal mechanism that applies a
Newton step on the model parameters that largely remove the inﬂuence of the deleted data points [50].
GKT is a zero-shot machine unlearning algorithm that imposes the constraint that zero training datais available to the unlearning algorithms [19].
Two recent studies propose online machine unlearningmethods for linear regression models [77] and linear support vector machine models [16], for further improving the efﬁciency of machine unlearning.
The former adapts users’ requests to delete theirdata before a speciﬁc time bar.
The latter conducts only the task of variable support vector machine.
Despite achieving remarkable success, most of the above machine unlearning methods consist of twosequential operations: (1) Training: train a model on the complete training data and (2) Unlearning:generate an unlearning model from the former.
The combination of two operations is computationallyexpensive when training complex models over large datasets.
In addition, they often sequentiallyredo the unlearning operations one by one, when addressing a series of machine unlearning requests.
Randomized smoothing has achieved the state-of-the-art certiﬁed robustness guarantees againstworst-case attacks by smoothing with isotropic Gaussian distribution [68, 22, 69, 75, 100, 71, 137,59, 5, 39, 134, 90, 86, 34, 12, 135, 113, 1, 15, 54, 84].
Specially, it takes a base classiﬁer f ( x ) as aninput, and outputs a smooth classiﬁer g ( x ) = argmax c ∈Y
P ε ∼D ( f ( x + ε ) =
c ) by averaging its predictionover isotropic Gaussian noise D = N (cid:0) 0 , σ 2 I (cid:1) of the input data x within its neighborhood ε .
Itprovides a tight certiﬁed robustness guarantee: g ( x ) can always return the most probable class c A ,i.e., g ( x + δ ) =
c A for all (cid:107) δ (cid:107) 2 < R , as long as the perturbation δ is within the certiﬁed radius R .
This motivates us to establish a connection between randomized smoothing for certiﬁed robustness onclassiﬁcation and randomized smoothing for certiﬁed machine unlearning on gradient quantization.
We analogize the data removals on the entire training data (i.e., the perturbations on the entire data)in
the machine unlearning to the adversarial attacks (i.e., the perturbations on the data samples) inthe certiﬁed robustness and liken the output quantized gradients in the former to the output discreteclass labels in the latter.
Since the output class labels in the latter through randomized smoothing areable to keep unchanged and correct against adversarial attacks within the certiﬁed radius, it is highlypossible that the output quantized gradients in the former through randomized smoothing can keepunchanged against data removals within the certiﬁed budget, which implies that the learnt modelshares the same gradients (and parameters) with the naive one retrained on only the remembered data.
Instead of performing two sequential operations of training and unlearning, this work directly trainsan unlearning model in advance, without the knowledge of the forgotten data, based on randomizeddata smoothing and gradient quantization.
We propose to execute the randomized smoothing onthe average x of data samples.
When there are data removal requests in the training data, x will beupdated with a new average x (cid:48) , which can be treated as a perturbed version of x .
As the class labelslie in a (countable) small discrete space but the gradients lie in a large continuous space, we proposea gradient quantization technique to produce discrete gradients in a three-class space {− 1 , 0 , 1 } .
The randomized smoothing method needs to sample a large number of points surrounding the inputdata for producing high conﬁdence certiﬁcates, e.g., 10 5 samples for 99 . 9% conﬁdence [22].
Inour context, the outputs of randomized smoothing are high-dimensional gradients.
We propose toutilize the Taylor expansion to approximate the output gradients of the sampled points for avoidingexpensive gradient computation.
We theoretically derive the error introduced by the Taylor expansion,the certiﬁed radius R regarding the perturbation surrounding ¯ x and the certiﬁed budget B of dataremovals (i.e., the maximally allowed amount of escaped data samples).
Notice that the response to the data removals is to erase the data samples and their associated labelstogether.
It is necessary to smooth on both of them.
However, for a given dataset, the relationshipbetween the samples and labels is a multi-valued non-continuous mapping, which is a non-integrablefunction.
This results in the dilemma of producing high conﬁdence certiﬁcates, even with samplingand estimation techniques.
Therefore, we propose a feasible solution based on randomized gradientsmoothing and gradient quantization.
We theoretically demonstrate the certiﬁed radius R (cid:48)
regardingthe gradient perturbations.
Most importantly, we recognize the correlations between two types ofradii R and R (cid:48) and between B and R , which are used to derive the certiﬁed budget B (cid:48) about R (cid:48) .
Wefurther integrate the model training, randomized gradient smoothing, and gradient quantization into auniﬁed framework for directly training a machine unlearning model with the data removal certiﬁcatesas a guidance, for guaranteeing that the model parameters and gradients keep unchanged against thedata removals within the certiﬁed budget.
In comparison with existing machine unlearning techniques, our randomized gradient smoothing andgradient quantization method exhibits three compelling advantages: (1) It simultaneously executes thetraining and unlearning operations, which is able to dramatically improve the unlearning efﬁciencyfor complex models on large-scale data; (2) The one-time operation of simultaneous training andunlearning can provide the timely response to a series of machine
unlearning requests, as long as the actual data removals are below the certiﬁed budget of data removals; and (3) It is agnostic to theremoved/forgotten data before performing the unlearning operation.
Empirical evaluation on real datasets demonstrates the superior performance of our PCMU modelagainst several state-of-the-art machine unlearning methods on image classiﬁcation.
More experiments, implementation details, and hyperparameter setting are presented in Appendices A.6-A.8.
Background
2.1 Randomized Smoothing for Certiﬁed Robustness
Randomized smoothing aims to build a smoothed classiﬁer g from a base classiﬁer f that maps inputsx ∈ R d to classes c ∈ C .
[Equation]
102 where D = N (cid:0) 0 , σ 2 I (cid:1) is a Gaussian probability distribution in R d for randomized smoothing.
greturns whichever class f is most likely to return when x is perturbed by noise ε .
Let p c ( x ) be the output probability of f over class c , i.e., p c ( x ) =
P ε ∼D ( f ( x + ε ) =
c ) .
Without lossof generality, we assume that p A ( x ) and p B ( x ) are the probabilities on the most probable class c
A andthe runner-up class c B respectively.
If P ( f ( x + ε ) =
c A ) ≥ p
A ≥ p B ≥ max c (cid:54) =
c A P ( f ( x + ε ) = c ) ,where p A ( x ) is a lower bound of p A ( x ) and p B ( x ) is an upper bound of p B ( x ) , then g ( x + δ )
=
c Afor ∀ δ ∈ R d , (cid:107) δ (cid:107) 2 ≤ R .
In this case, the smoothed classiﬁer g can always output the correctprediction as long as the perturbation δ is within a certiﬁed l 2 -norm radius of R .
Theorem 1.
Let f : R d → Y be any deterministic or random function, and let ε ∼ N (cid:0) 0 , σ 2 I (cid:1) .
Letg be deﬁned as in (1).
Suppose c A ∈ Y and p A , p B ∈ [0 , 1] satisfy [22]:
[Equation]
Then g ( x + δ ) =
c A for all (cid:107) δ (cid:107) 2 < R , where
[Equation]
115 where Φ − 1 is the inverse of the standard Gaussian CDF.
2.2 Machine Unlearning
Machine unlearning aims to enable the trained models to forget what has been learned from the datato be removed.
Speciﬁcally, given a training dataset of N samples D = { x i , y i } Ni =1 .
Each samplex i ∈ R d is associated with a label y i ∈ Y
= { 1 , 2 , ..., Y } , where Y is the number of classes.
Aclassiﬁcation model M ( D ) is trained on the complete training dataset D .
The users can submit a data removal request at any time.
Thus, the complete training data D ispartitioned into two subsets: D f ⊆ D denoting the data which we wish the classiﬁcation model toforget and D r ⊆ D specifying the data which we want the model to remember ( D = D f ∪ D r ).
The goal of machine unlearning is to unlearn the forgotten data D f , i.e., eliminate the inﬂuence of
D f from M ( D ) .
A straightforward solution is to use the remembered data D r as the training datato retrain a new classiﬁcation model M r ( D r ) from scratch.
However, this naive method is oftentime-consuming over large-scale datasets.
An efﬁcient algorithm is to directly generate a sanitizedmodel M u ( D, D f , M ) from the deployed model M ( D ) that approximates M r ( D r ) , i.e., M u ( D [Equation]
M
[Equation]
≈
[Equation]
r
[Equation]
Randomized Data Smoothing and Gradient Quantization
The idea of this work is to establish a connection between randomized smoothing for certiﬁedrobustness on classiﬁcation and randomized smoothing for certiﬁed machine unlearning on gradientquantization.
By leveraging the theory of randomized smoothing and gradient quantization, wetheoretically derive the certiﬁed radius R regarding the perturbation surrounding the data average ¯ xand the certiﬁed budget B of data removals.
The gradient G ( x, y ) ∈ R T of a machine learning model is given as follows. [Equation]
[Equation]
where L is the loss function, e.g., cross-entropy for image classiﬁcation.
w is the model parameter.
We propose to quantize each dimension t ( t = 1 , · · · , T ) of the continuous gradient G ( x
, y ) ∈
R Tover a discrete three-class space {− 1 , 0 , 1 } , for mimicking the classiﬁcation in the randomizedsmoothing for certiﬁed robustness.
[Equation]
143 where Q ( t ) maps a gradient dimension t to a three-dimensional vector [ −| t − σ 2 | , −| t | , −| t + σ 2 | ] ,where each component denotes the similarity score between t and − σ 2 , 0, or σ 2 .
σ is the standarddeviation of the Gaussian probability distribution in the randomized smoothing and also servesas a quantization threshold in our method.
The details of the selection of quantization thresholdare presented in Appendix A.2.
Therefore, all T gradient dimensions are partitioned into threeintervals: ( −∞ , − σ 2 / 2] that comes near to − σ 2 , [ − σ 2 / 2 , σ 2 / 2] that is closer to 0, and [ σ 2 / 2 , ∞ )that approaches σ 2 .
Each component in Q ( t ) with the Softmax function also represents the probabilityof the gradient dimension t belonging to classes -1, 0, or 1.
The most probable class c A ∈ {− 1 , 0 , 1 }in Q ( t ) is assigned to dimension t as a ﬁnal quantized gradient dimension.
We represent the composition F ( x, y ) of gradient computation and quantization as follows.
[Equation]
154 We use F t ( x, y ) to denote the output three-dimensional quantization vector of the t th ( t = 1 , · · · , T )dimension of the gradient G ( x, y ) and use F tc ( x, y ) to represent the c th ( c ∈ {− 1 , 0 , 1 } )
componentof F t ( x, y ) .
As the data removal is treated as the noise on the entire training data, we use the average ¯ x of all datasamples to represent the entire training data.
[Equation]
160 where ¯ y is the average of the class labels of all data samples.
The randomized data smoothing for certiﬁed unlearning on gradient quantization is deﬁned below.
[Equation]
163 where D = N (cid:0) 0 , σ 2 I (cid:1) is a Gaussian distribution.
S t is a smoothed version of a base gradientquantizatier F t that maps the t th gradient dimension about inputs (¯ x, ¯ y ) to gradient classes c ∈{− 1 , 0 , 1 } .
S t returns whichever class F t is most likely to return when (¯ x, ¯ y ) is perturbed by ( ε x , ε y ) .
The randomized smoothing method needs to sample a large number of points surrounding the inputdata for producing high conﬁdence certiﬁcates, e.g., 10 5 samples for 99 . 9% conﬁdence [22].
Inour context, computing the gradients for massive samples is extremely inefﬁcient.
We utilize the
Taylor expansion to approximate the output gradients of sampled points, based on the outputs fromthe original sample (¯ x, ¯ y ) .
For each quantization component F tc (ˆ¯ x, ˆ¯ y ) for a sample in a Gaussiandistribution surrounding (¯ x, ¯ y ) , where ˆ¯ x = ¯ x + ε x , ˆ¯ y
=
¯ y + ε y , ε x , ε y ∼ D , we take the Taylorexpansion of F t c ( ˆ ¯ x, ˆ ¯ y ) at (¯ x, ¯ y ) as follows.
[Equation]
174 where O j (ˆ¯ x, ˆ¯ y )
=
1 ( j +
[Equation]
{ (cid:80) · · · (cid:80) ∂
[Equation]
F tc ( ξ ) ∂x
[Equation]
1 − k (
[Equation]
x − ¯ x ) k
(ˆ¯ y − ¯ y )
j +1 − k } , ξ ∈ ((ˆ¯ x, ¯ x ) , (ˆ¯ y, ¯ y )) , j =3 , · · · , ∞ , and k
= 1 , 2 , 3 .
The following theorem derives the error introduced by the Taylor expansion.
Theorem 2.
The error introduced by the Taylor expansion of F t c ( ˆ ¯ x, ˆ ¯ y ) at (¯ x, ¯ y ) is
[Equation]
(11) 179 where
L j = max k
=1 , ··· ,j ∂ j h
i ∂x k ∂y j − k
and M is the number of sampled points.
Please refer to Appendix A.5 for detailed proof of Theorem 2.
Notice that
[Equation]
Thus, we derive the probabilities p A and p B on the most probable class c A and the runner-up classc B in the randomized smoothing for certiﬁed machine unlearning ( c A , c B ∈ {− 1 , 0 , 1 } ) , based onthe error introduced by the Taylor expansion.
[Equation]
[Equation]
188 Notice that the correlation between the data ¯ x and its classes ¯ y is ﬁxed for a given dataset.
We denotethis correlation as ¯ y
= H (¯ x ) where H : R d (cid:55)→ C .
Thus, the randomized smoothing for certiﬁedmachine unlearning in Eq.(9) is rewritten as an equivalent one as follows.
[Equation] [Equation]
192 where ˜ F t (¯ x ) =
F t (¯ x, H (¯ x )) =
F t (¯ x, ¯ y ) .
Based on the computed p A , p B , and F (cid:48) (¯ x ) , we can obtain the certiﬁed radius R regarding theperturbation surrounding ¯ x and the certiﬁed budget B of data removal.
Theorem 3.
Let ε ∼ D
=
N (cid:0) 0 , σ 2 I (cid:1) and ˜ S t (¯ x ) =
argmax c ∈{− 1 , 0 , 1 }
P ε ∼D ( ˜ F t (¯ x + ε ) =
c ) .
Suppose thatfor a speciﬁc ¯ x ∈ R d , there exist c A ∈ {− 1 , 0 , 1 } and p A , p B ∈ [0 , 1] such that:
[Equation]
198 Then ˜ S t (¯ x + δ ) =
c A for all (cid:107) δ (cid:107) 2 < R , where
[Equation]
200 where Φ − 1 is the inverse of the standard Gaussian CDF.
Theorem 4.
Let R be the certiﬁed radius of ¯ x ∈ R d based on ˜ S t (¯ x ) = argmax
[Equation]
then the certiﬁed budget of data removals is
[Equation]
204 Please refer to Appendix A.5 for detailed proof of Theorems 3 and 4.
The above method is effective for certiﬁed machine unlearning, but it is computationally expensiveto calculate high-dimensional double integrals in p A and p B .
We can reduce the double integrals tothe single integrals through ¯ y
= H (¯ x ) .
However, H is essentially a multi-valued non-continuousmapping, which is not an integrable function and makes the above method impractical.
[Equation]
Randomized Gradient Smoothing and Quantization
In order to avoid the dilemma of computing practical p A , p B and R , we propose a feasible solutionbased on randomized gradient smoothing and gradient quantization.
We theoretically demonstratethe certiﬁed radius R (cid:48) regarding the gradient perturbations.
We recognize the correlations between Rand R (cid:48) and between B and R , which are used to derive the certiﬁed budget B about R (cid:48) .
We ﬁrst calculate the gradient G ( x i , y i ) in terms of each sample ( x i , y i ) and the gradient average ¯ G .
[Equation]
217 The randomized gradient smoothing for certiﬁed unlearning on gradient quantization is deﬁned below.
[Equation]
[Equation] 219 where D = N (cid:0) 0 , σ 2 I (cid:1) is a Gaussian distribution. S t (cid:48) is a smoothed version of a base gradientquantizatier Q t that maps each dimension t of the gradient ¯ G to gradient classes c ∈ {− 1 , 0 , 1 } .
S t (cid:48)returns whichever gradient class Q t is most likely to return when ¯ G is perturbed by noise ε .
We compute the probabilities over three intervals of ( −∞ , − σ 2 / 2] , [ − σ 2 / 2 , σ 2 / 2] , and [ σ 2 / 2 , ∞ ) .
[Equation]
224 Now, we can directly generate the corresponding probabilities p (cid:48)
A and p (cid:48) B .
[Equation] [Equation]
226 By following the similar strategy in Theorem 3, we can derive the corresponding certiﬁed radius R (cid:48) .
Namely, S t (cid:48) ( ¯ G + δ ) =
c A for all (cid:107) δ (cid:107) 2 < R (cid:48) , where
[Equation] [Equation]
However, it is difﬁcult to obtain the corresponding certiﬁed budget of data removal from R (cid:48) , since
R (cid:48) is related to the perturbations over the gradient ¯ G , instead of the data ¯ x .
The following theoremdemonstrates the correlation between two types of radii R and R (cid:48) .
Theorem 5.
Let R and R (cid:48) be the certiﬁed radii of the above two algorithms respectively and L bethe Lipschitz constant of gradient G ( x, y ) ∈ R T , then [Equation]
[Equation]
235 By combining Theorems 4 and 5 together, we derive the certiﬁed budget B (cid:48) of data removal from R (cid:48) .
[Equation]
[Equation]
237 In addition, we conduct the convergence analysis of our prompt certiﬁed machine unlearning algorithmbased on randomized gradient smoothing and quantization.
Theorem 6.
Let S t (cid:48) ( ¯ G ) be the randomized gradient smoothing for certiﬁed machine unlearning ongradient quantization, L , L 1 , and L 2 be the Lipschitz constants of G , Q t , and S t (cid:48) respectively, i.e.,
[Equation]
any
[Equation]
[Equation]
242 If we run gradient descent for k iterations with a ﬁxed step size s ≤ 1 L 2 L 1 L , it will yield a solution
S t (cid:48) ( k ) which satisﬁes
[Equation]
(28) 245 where S t (cid:48) ( q (0) ) is the initial solution and S t (cid:48) ( q ∗ ) is the local optimal solution.
This means that gradient descent is guaranteed to converge and that it converges with rate O (1 /k ) .
Please refer to Appendix A.5 for detailed proof of Theorems 5 and 6.
Finally, we integrate the model training for a speciﬁc learning task (e.g., image classiﬁcation)randomized gradient smoothing, and gradient quantization into a uniﬁed framework for directlytraining a machine unlearning model with the data removal certiﬁcates as a guidance, for guaranteeingthat the model parameters and gradients keep unchanged against the data removals within the certiﬁedbudget.
The corresponding parameter update is given below.
[Equation]
29) 254 where w is the model parameter and η is a learning rate.
Experimental Evaluation
In this section, we have evaluated the effectiveness of our PCMU model and other comparison methodsfor machine unlearning over three popular image classiﬁcation datasets: Fashion-MNIST [131, 51,38], CIFAR-10 [65, 44, 45, 115, 51, 38], and SVHN [93, 50, 6].
We train the classiﬁers on the trainingset and test them on the test set for three datasets.
We train a convolutional neural network (CNN) on
[Table] [Table]
Fashion-MNIST for clothing classiﬁcation.
We train LeNet over CIFAR-10 for image classiﬁcation.
We apply the ResNet-18 architecture on SVHN for street view house number identiﬁcation.
Weevaluate the performance of various machine unlearning methods on three datasets with the ratioof data removal between 5% and 20%.
In this work, by following several representative machineunlearning methods [6, 51, 128], where each learning request is modeled as a random draw from thetraining data in terms of a uniform distribution.
Given a ratio of data removal, the forgotten data D fare sampled uniformly from the complete training data D with this ratio.
The remaining dataset D r(i.e.
, D = D f ∪ D r ) will be considered as the remembered data. This sampling approach is morerealistic since a removal request may be applied to any data examples with the same probability. Baselines.
We compare the PCMU model with nine state-of-the-art machine unlearning models.
Fisher is a scrubbing procedure that removes information from the trained weights, without the needto access the original training data, nor to retrain the entire network [44].
certiﬁed removal providesa strong theoretical guarantee that a model from which data is removed cannot be distinguishedfrom a model that never observed the data to begin with [50].
DeltaGrad is a rapid retrainingmachine learning model based on information cached during the training phase [129].
NTK removesdependency on a cohort of training data from a trained deep network that improves upon andgeneralizes previous methods to different readout functions [45].
Unrolling SGD is a taxonomy ofapproximate unlearning which concludes with veriﬁcation error as a metric to study as it subsumes alarge class of unlearning criteria [115].
SISA is a practical approach for unlearning that relies on datasharding and slicing to reduce the computational overhead of unlearning [6].
Adaptive Unlearninggives a general reduction from deletion guarantees against adaptive sequences to deletion guaranteesagainst non-adaptive sequences.
FedEraser is a federated unlearning methodology that caneliminate the inﬂuences of a federated client’s data on the global model while signiﬁcantly reducingthe time consumption [78].
MCMC unlearning designs an MCMC inﬂuence function to characterizethe knowledge learned from data, which then delivers the MCMC unlearning algorithm
To ourbest knowledge, this work is the ﬁrst to execute one-time operation of simultaneous training andunlearning in advance for a series of machine unlearning requests.
Variants of PCMU model.
We evaluate two versions of PCMU to show the strengths of differenttechniques.
PCMU-N uses the basic model with only the gradient quantization.
PCMU operates withthe full support of both the randomized gradient smoothing and the gradient quantization techniques.
Notice that the gradient quantization is a necessary operation to convert continuous gradients todiscrete gradient classes for the randomized gradient smoothing in our PCMU model.
Thus, wecannot validate the version with only the randomized gradient smoothing.
Evaluation metrics.
By following the same settings in several representative machine unlearning models [44, 45, 115, 38], we use four popular measures in machine unlearning to verify theperformance of different methods: Accuracy , Error f (classiﬁcation errors on the forgortten data
D f ), Error r (errors on the remembered data D r ), and Error t (errors on the test data).
Since themodel M r ( D r ) ( Retrain ) that uses only the remembered data D r as the training data retrained fromscratch has never seen the forgotten data D f , it is usually used as the gold standard for evaluating theperformance of machine unlearning algorithms [44, 38].
Ideally, the accuracy and three errors of theunlearning models should match that of the retrained model M r ( D r ) .
Machine unlearning accuracy with varying ratios of data removal.
Tables 1-4 exhibit the accuracy obtained by eleven machine unlearning approaches by varying the ratio of unlearning request /data removal between 10% and 20%.
Retrain represents the model retrained on only the remembereddata D r from scratch, without the knowledge of the forgotten data D f .
A machine unlearningalgorithm with more similar performance to the Retrain model achieves a better unlearning result. [Table]
[Table] [Table] It is observed that among ten approaches except the Retrain model, no matter how large the ratiosof data removal are, the PCMU method achieves the closest accuracy to the Retrain model in alltests, showing the effectiveness of PCMU to the machine unlearning.
Compared to the absoluteperformance difference between other baselines and the Retrain model, PCMU, on average, achievesat least 5.56% and 15.17% improvement of absolute accuracy difference on Fashion-MNIST and
CIFAR-10 respectively.
Notice that the accuracy and error on test data by our PCMU keep unchangedsince it performs one-time operation of simultaneous training and unlearning for addressing multipleunlearning requests.
In addition, the promising performance of PCMU over Fashion-MNIST and
CIFAR-10 implies that PCMU has great potential as a general machine unlearning solution to otherimage datasets, which is desirable in practice.
Machine unlearning error with varying ratios of data removal.
Tables 1-4 also show the classiﬁcation errors on the deleted data D f ( Error f ), errors on the remembered data D r ( Error r ),and errors on the test data ( Error t ) by eleven machine unlearning methods respectively.
We haveobserved that the performance of our PCMU method behaves similarly and achieves at least 13.73%and 16.01% boost of absolute error difference on two datasets respectively.
PCMU substantiallyoutperforms the performance of other baselines in most experiments, especially on the CIFAR-10dataset.
In addition, the errors by our PCMU are not sensitive to the ratio of data removals.
This isbecause that our PCMU method performs one-time operation of simultaneous training and unlearningwhen addressing a series of machine unlearning requests, as long as the ratio of actual data removalsis below the certiﬁed budget of data removals in our PCMU.
However, other baselines need tosequentially handle these machine unlearning requests one by one.
Ablation study.
Figure 1 exhibits the unlearning performance with the Retrain model and twovariants of the PCMU model over two datasets of Fashion-MNIST and CIFAR-10 respectively.
We have observed the exact PCMU achieves the closest accuracy and errors to the Retrain modelover two datasets, which are obviously better than PCMU
-N. A reasonable explanation is that our
PCMU method utilizes the randomized gradient smoothing and gradient quantization techniques forsupporting certiﬁed machine unlearning.
It further uses the certiﬁcates as a guidance to train themachine unlearning model, for guaranteeing that the model parameters and gradients keep unchangedagainst the data removals within the certiﬁed budget.
Running time.
Tables 1-4 report the running time achieved by all comparison methods over twodataset to produce machine unlearning results respectively.
We observe that PCMU scales wellwith deep neural network architectures over different image datasets and shows good efﬁciency formachine unlearning.
Our PCMU method achieves better efﬁciency than most baseline methods, [Table] [Figure] [Figure]
except DeltaGrad and Unrolling SGD.
As discussed above, our PCMU method performs one-timeoperation of simultaneous training and unlearning when addressing a series of machine unlearningrequests.
However, DeltaGrad and Unrolling SGD need to sequentially handle these machineunlearning requests one by one.
This is clearly a computationally expensive process when the numberof machine unlearning requests is huge.
Impact of standard deviation.
Figure 2 (a) measures the performance effect of standard deviationof the Gaussian distribution in the randomized smoothing for machine unlearning by varying σ from0.025 to 0.3.
Notice that the Retrain and PCMU-N models do not contain the module of randomizedsmoothing.
Thus, their accuracy scores keep unchanged with varying σ .
We have witnessed theperformance curves by PCMU initially increase quickly and then become stable or even slight dropwhen σ continuously increases.
Initially, a large σ can help utilize the strength of randomized gradientsmoothing and quantization for directly training a machine unlearning model in advance.
Later on,when σ continues to increase and goes beyond some thresholds, the performance curves becomestable.
A rational guess is that after the data removals have been already certiﬁed at a certain thresholdand considered in the training of machine unlearning models, our PCMU model is able to generate agood machine unlearning result.
When σ continuously increases, this does not affect the performanceof machine unlearning any more.
Impact of data removal ratio.
Figure 2 (b) evaluates the accuracy impact of data removal ratiosvarying from 5% to 20% on three datasets of Fishion-MNIST, CIFAR-10, and SVHN.
It is observedthat when changing data removal ratios, the accuracy by our PCMU model matchs well with thatof the retrained model from scratch.
The performance by our PCMU model keeps relatively stable,since our method directly trains a unlearning model based on the certiﬁed budget of data removals inadvance and performs one-time operation of simultaneous training and unlearning, as long as theratio of actual data removals is below the certiﬁed budget of data removals.
Conclusions
In this work, we have proposed a prompt certiﬁed machine unlearning algorithm that executes onetime operation of simultaneous training and unlearning in advance.
First, we establish a connectionbetween randomized smoothing for certiﬁed robustness on classiﬁcation and randomized smoothingfor certiﬁed machine unlearning on gradient quantization.
Second, we propose a certiﬁed machineunlearning model based on randomized data smoothing and gradient quantization.
Finally, we presentanother practical framework of randomized gradient smoothing and quantization, due to the dilemmaof producing high conﬁdence certiﬁcates in the ﬁrst framework.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope
[Yes] See Sections 1, 3, and 4.
(b) Did you describe the limitations of your work?
[Yes] See Section A.9.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See [Table] Section A.9.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
[Table]
them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section A.5.
[Table]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section A.5. [Table]
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We includethe citations and URLs of all datasets used in this work and all codes of third-partybaselines in Sections 5 and A.8.
Since the datasets used are all public datasets and our
[Table] methodologies and the hyperparameter settings are explicitly described in Section 3,4, 5, and A.8, our codes and experiments can be easily reproduced on top of a GPU server.
We promise to release our open-source codes on GitHub and maintain a projectwebsite with detailed documentation for long-term access by other researchers and end-users after the paper is accepted. [Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section A.8.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section A.8.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sections 5 andA.8.
[Table]
(b) Did you mention the license of the assets?
[Yes] See Section A.8.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Table]
[No ] Since the datasets used are all public datasets and our methodologies and thehyperparameter settings are explicitly described in Section 3, 4, 5, and A.8, our codesand experiments can be easily reproduced on top of a GPU server.
We promise torelease our open-source codes on GitHub and maintain a project website with detaileddocumentation for long-term access by other researchers and end-users after the paperis accepted.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]