No Matter How You Slice It: Machine Unlearning with SISA Comes at the Expense of Minority Classes
Abstract —Machine unlearning using the SISA technique promises a significant speedup in model retraining with only minor sacrifices in performance.
Even greater speedups can be achieved in a distribution-aware setting, where training samples are sorted by their individual unlearning likelihood.
Yet, the side effects of these techniques on model performance are still poorly understood.
In this paper, we lay out the impact of SISA unlearning in settings where classes are imbalanced, as well as in settings where class membership is correlated with unlearning likelihood.
We show that the performance decrease that is associated with using SISA is primarily carried by minority classes and that conventional techniques for imbalanced datasets are unable to close this gap.
We demonstrate that even for a class imbalance of just 1:10, simply down-sampling the dataset to a more balanced single shard outperforms SISA while providing the same unlearning speedup.
We show that when minority class membership is correlated with a higher- or lower-than- average unlearning likelihood, the accuracy of those classes is (cid:58)(cid:58)(cid:58) can (cid:58)(cid:58) be (cid:58) either improved or diminished in distribution-aware SISA models.
This relationship makes the model sensitive to naturally occurring unlearning likelihood correlations. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) While (cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) models (cid:58)(cid:58)(cid:58) tend (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sensitive [Table]
to
[Table] class [Table] distribution (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) found (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) impact (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) imbalanced (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroups (cid:58)(cid:58) or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) model (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) fairness.
(cid:58)
Our work contributes to a better understanding of the side effects and trade-offs that are associated with SISA training
Index Terms —machine unlearning, class imbalance (cid:58) , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) fairness (cid:58)
I. I NTRODUCTION
Under legislation such as the California Consumer Privacy
Act (CCPA) 1 in California, the General Data Protection Regulation (GDPR) 2 in the European Union, or Personal Information Protection and Electronic Documents Act (PIPEDA)in Canada, people have a right to request the deletion of their personal data, also referred to as right to be forgotten .
While this right can often be honored easily by deleting all database entries relating to a specific person, erasure becomes much more difficult if the characteristics of ones personal data have already been ingrained in a trained machine learning model. Shard Shard Shard
Shard
Shardrandom assignment to shards and slices Shard Shard Shard
Shard
Shardadaptive assignment into fewest possible shards
Shard
Shard
Shard
Shard
Shardadaptive assignment into last trained slices
[Figure]
The impossibility to separate training data from trained models is exemplified by membership inference attacks [1], which allow an adversary to determine if a given data record was part of the training set of a black-box model.
This inseparability motivates the need for machine unlearning [2], [3], which refers to any technique that is able to remove the influence that any particular training point had on the final model.
The term exact unlearning refers to methods that formally remove the data points influence by retraining the model without it, while approximate unlearning refers to methods that try to approximate the model parameters that exact unlearning would yield without actually retraining the model [4].
Retraining a model from scratch is time-, energy- and costintensive, especially when the model has many parameters or the training dataset is large.
One exact unlearning method that tries to reduce the computational overhead associated with retraining is SISA [5], which stands for Sharded, Isolated,
Sliced, and Aggregated training.
SISA training divides the training data into n (cid:58) S (cid:58) disjoint shards of approximately equal size.
The data in each shard is then further separated into m (cid:58) R (cid:58) disjoint slices.
During training, one constituent model is trained per shard.
In a single shard, this means training a model on the first slice and then saving the parameters of the model.
The model parameters are then loaded again for each of the remaining slices, further trained, and saved again, until the training data in the last slice has been processed.
These last obtained parameters define the final constituent model of that shard.
Atinference time, the input is processed by each constituent model and the responses are then aggregated, for example through majority voting [5].
If any given data point has to be deleted, only the constituent model associated with the shard containing the data point has to be modified.
Retraining only has to take place for the slice containing the data point as well as all following slices in the same shard, and the last saved checkpoint before those slices can be used as starting point.
For a single data point, retraining is quicker if it is located in a later slice.
If multiple data points have to be removed at once, it is beneficial if those data points are located in the same shard.
This effectively limits the total cost of retraining to the maximum individual retraining cost, as all following slices have to be retrained anyways.
By default, data points in SISA training are assigned to shards and slices at random [5].
Even if the number of deletion requests processed at a time is relatively low, this makes it likely that all constituent models will have to be retrained.
However, if the unlearning likelihood of individual data points is either known in advance or can be estimated with reasonable accuracy, the authors [5] suggest that this would allow us to group high-likelihood data points together to further improve the cost of retraining.
This can be realized by placing data points with a high unlearning likelihood in designated shards, which the authors call distribution-aware sharding .
Alternatively, and motivated by the observation made in the previous paragraph, high-likelihood data points can also be preferably placed in later slices, which we call distribution-aware slicing .
Examples of the resulting shard and slice composition of all strategies can be seen in Fig.
As the complete model parameters have to be saved n ∗ m (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) S ∗ R (cid:58) times instead of just once, SISA effectively represents a trade-off between disk space and (cid:58)(cid:58)(cid:58) (and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) inference (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) time) (cid:58)(cid:58)(cid:58) and computing effort for retraining [5].
As (cid:58)(cid:58)(cid:58) long (cid:58)(cid:58)(cid:58) as (cid:58) disk space is considerably cheaper than GPU compute (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) there (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) much (cid:58)(cid:58)(cid:58)(cid:58) more (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] than [Table]
inference [Table] steps, this trade-off is beneficial. The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) retraining (cid:58) speedup associated with using SISA ranged between 1 .
× and 4 .
63 × in the experiments conducted in the original publication [5] for reasonable amounts of shards and slices.
At the same time, using SISA was associated with a decline in model accuracy.
The authors discussed that the speedup of retraining is mainly dependent on the number of shards, and increasing the number of slices has a quickly diminishing return beyond a few slices [5].
However, the same holds true for the accuracy degradation of the final model, which gets bigger if more shards are used.
The explanation given by the authors is that each constituent model must be presented with sufficiently many data points during training in order to reach a good performance [5].
A single constituent model sees the exact same amount of training samples for any number of slices – only in a different order.
However, when the number of shards is increased, the absolute amount of training samples per constituent model sinks proportionally.
This impairs each constituent models ability to generalize and thus lowers the accuracy of the entire ensemble.
This observation motivates the question how SISA models behave when the number of shards is low and the absolute number of training examples per shard is high, but the number of training examples for individual underrepresented class
[Table] classes is low. If a magic threshold for good constituent model generalization exists, can it be exceeded by one class, but fallen short of by another (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sample (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) numbers (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) across (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classes (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reduced (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) linearly (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shard, (cid:58)(cid:58)(cid:58)(cid:58) will (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) neural (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) scaling (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) laws (cid:58)(cid:58)(cid:58)(cid:58) harm (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) performance (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]
In summary, the contributions of this paper are:
• We show that the accuracy gap introduced by SISA training is bigger for minority classes, and gets bigger as the imbalance ratio
rises • We demonstrate that both simple and advanced methods against class imbalance, even though successful in bringing the performance of majority and minority classes closer together, are unable to eliminate this unequal burden.
• We show that SISA is outperformed by a simple downsampled model with 1 shard on minority classes while preserving the same retraining speedup.
• We demonstrate that minority classes are sensitive to correlations with unlearning likelihood in distributionaware SISA settings and that a positive (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) positive (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) negative correlation between unlearning likelihood and minority class membership can (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) used (cid:58)(cid:58) to (cid:58) improve the performance of said classes. • (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58) show
(cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) models (cid:58)(cid:58) do (cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) introduce (cid:58)(cid:58)(cid:58) an (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) equivalent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accuracy (cid:58)(cid:58)(cid:58) gap (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) imbalanced (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroups, (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58) have (cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) impact (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] II. R ELATED W ORK
The topic of Machine Unlearning [2], [3] or the removal of data from trained models is a recent topic in machine learning [4], [6]–[9].
One framework for unlearning is SISA training (Sharded, Isolated, Sliced, and Aggregated training) [5].
SISA is discussed in literature as a tool for data protection and trustworthy AI [10]–[12].
Besides allowing effective machine unlearning, SISA seems to protect well from attacks on privacy as the aggregation step of SISA reduces the influence of individual samples on the final prediction [13].
However, it is theoretically possible for an adversarial actor to completely degrade the accuracy of SISA [14].
Another topic related to machine learning are imbalanced classes [15], [16].
Classes are imbalanced if the size of at least one class (minority class) is considerably smaller than another class (majority class).
As a result, the performance of the model on the minority class has a smaller impact on the average accuracy on the dataset, which in return leads to solutions that classify minority classes inaccurately.
According to [17], most existing solutions for the problem of imbalanced classes include introducing learning bias against majority classes, under- or oversampling of data, and costsensitive learning.
Overviews of current methods to tackle class imbalance can be found in [18], [19].
To the best knowledge of the authors, no work has yet investigated the impact of class imbalance on SISA learning or machine unlearning.
III. H OW D OES SISA A FFECT I MBALANCED D ATASETS ?
In many applied contexts, different classification outcomes are not equally likely.
One such context where the processed data is both highly imbalanced and highly sensitive is the medical domain (for examples, see [20]–[22]).
If you intend to build a classifier that is able to detect the presence of a rare disease, you will most likely have to work with many samples from healthy patients and only very few samples from sick patients.
For many tasks, imbalances of 1:100 and beyond are not unheard of (e.g. [21]).
At the same time, medical datasets, especially image datasets, can be very large and computationally expensive to process.
For example, fullscale histopathology images can reach the size of gigapixels [23] and models working with these images take long to train even when using helpful techniques such as downsampling and tiling.
If the training data for such models fall under a privacy-focused jurisdiction, using SISA seems like a good idea to ensure that retraining is both faster and cheaper.
But does SISA impact imbalanced datasets differently?
A. Datasets
When SISA was introduced [5], the effects on the performance were evaluated on the MNIST [24], Purchase [25],
SVHN [26], CIFAR-100 [27], Imagenet [28], and MiniImagenet [29] dataset.
The authors assigned task complexities to each of these datasets, with the first three being regarded as easy and the latter three as hard tasks.
According to the authors, sharding and slicing combined have no significant impact on accuracy for easy tasks, and result in only a small accuracy decrease in the single percentage point realm for hard tasks when pretraining is used [5].
When comparing the ratio of the largest and smallest class in each dataset (see Tab. I), the maximal imbalance ratio for the evaluated datasets ranges between 1:1 and 1:5.2.
In the original paper [5], results were reported in terms of overall average
[Table]
model accuracy, so even for the datasets where imbalances were present, the accuracies of individual classes were not reported separately.
In order to evaluate the impact SISA has on imbalanced classes, we are using the EMNIST [30] (“Extended MNIST”) dataset, a superset of MNIST containing handwritten digits and letters.
More specifically, we are utilizing the EMNIST
Digits dataset, a balanced dataset containing 24,000 training and 4,000 testing samples per class.
Having a dataset multiple times bigger than MNIST allows us to explore the limits of
SISA to a larger extent, and create synthetic class imbalances up to 1:1000.
Even though the SVHN dataset also has a large total size and 10 classes [26], the fact that house number digits do not occur with an equal probability gives this dataset a natural imbalance, making it more difficult to introduce specific imbalances in an experimental setting.
B. Learning with Class Imbalance
Many methods for learning with class imbalance have been proposed over the years, which can roughly be divided into data-level methods, algorithm-level methods and hybrid methods [18].
The following methods are utilized in our paper:
• data-level methods : random over-sampling (ROS), random under-sampling (RUS) • algorithm-level method : cost-sensitive learning, focal loss, label-distribution-aware margin (LDAM) loss
All methods used in our experiments require knowledge of the global class distribution.
However, when using SISA, shards and slices are isolated, and we only have local knowledge reflecting the class distribution in the current slice.
Because of this, the used sample numbers and class costs in our experiments are recalculated for each slice, and will differ from slice to slice.
If any individual shard has a class distribution substantially different from the overall dataset, the sample numbers and class costs will reflect that difference and differ substantially as well.
Both ROS and RUS [31], [32] work by manipulating the dataset.
In ROS, random copies of minority samples are added to the dataset until all classes are of equal size.
In RUS, random samples from the majority classes are removed from the dataset until all classes are of equal size.
In the context of machine unlearning, ROS is undesirable, as it increases the size of the dataset and thus extends the retraining time.
RUS is desirable, as it decreases the size of the training dataset and thus shortens the retraining time.
Because of this, RUS can in itself be considered a method against class imbalance that comes with improved machine unlearning as a free add-on.
To achieve an equivalent speedup as a SISA model with n (cid:58) S (cid:58) shards, RUS has to result in a final dataset size that is smaller than the original by a factor of √ n (cid:58)(cid:58)(cid:58) √ S .
This relationship can be derived from the fact that a 1 /n (cid:58)(cid:58)(cid:58) 1 /S (cid:58) monolithic baseline model is always faster than a SISA model with n (cid:58) S (cid:58) shards by a factor of n (cid:58)(cid:58) S (cid:58) [5].
Because of this, we will not combine RUS with SISA, but rather evaluate it as a third kind of machine unlearning method that can be combined with additional algorithm-level methods against class imbalance.
In order for the mentioned (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) speedup equivalence to hold, the RUS model has to be sliced the same way as the SISA model .
(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unlearning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) requests (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) have (cid:58)(cid:58) to (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) processed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) individually.
(cid:58)(cid:58)
If (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unlearning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) requests (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) processed (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) batches (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) speedup (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) model (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) degrades (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) quicker (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)
[Table] Cost-sensitive learning [33] allows us to assign a different importance or cost to each class.
Then, during training, instead of minimizing our prediction error, we are minimizing the costs of our prediction errors.
While cost-sensitive learning can be used to achieve a number of goals [33], the intention for using it with imbalanced datasets is to produce models that have more similar accuracies on majority and minority classes, despite substantial differences in the number of training examples.
In our experiments, we are combining the loss with class-wise costs, where the costs are equal to the square root of the inverse number of samples per class, normalized by the average number of samples per class.
This means that a perfectly balanced dataset has class costs of only ones, whereas any imbalanced dataset will have class costs above one (minority classes) and below one (majority classes).
Thus, any individual sample belonging to a minority class will have a larger effect on the batch loss than a sample belonging to a majority class, which counteracts the lower expected absolute number of such samples in each batch.
Focal loss [34] was originally designed for dense object detection, where a classifier was moved over many locations of an image.
However, this means that many locations have no object and only a few include the target object.
If all locations are used for training this generates a highly imbalanced dataset.
To counter this effect, Lin et al.
[34] added a modulating term to cross-entropy loss which gives wellclassified samples a lower influence on the average batch loss than hard misclassified samples.
The scaling factor γ in the modulating term controls how much the impact of easy samples is reduced.
LDAM loss [35] focuses on the decision boundary of the classifier and forces the model to have a larger margin between the decision boundary and minority classes than between the decision boundary and majority classes.
This allows the model to achieve better generalization for minority classes while losing only minor generalization on the majority class.
Both focal loss and LDAM loss can be combined with costsensitive learning and have been shown to complement each other [35].
In our experiments, they are thus combined with the costs described in the section above.
C. Implementation details
To find out which effect SISA has on majority and minority classes, we establish an experimental regime that allows us to measure the impact of SISA on prediction performance depending on the imbalance of each class.
Out of the 10 EMNIST [30] digit classes, we keep 7 classes as they are, and keep samples from the 3 remaining classes with a probability of 10%, 1% and 0.1%, corresponding to resulting class imbalances of 1:10, 1:100 and 1:1000.
Each experiment is repeated 10 times in a way that ensures that each digit gets assigned each of the three imbalance ratios exactly once.
This ensures that classes which are easier or harder from the beginning (such as discerning the digits 3 and 8) have no effect on the reported results.
The used model architecture is a ResNet-18 [36] which was modified to process black-and-white instead of RGB images and is followed by a classifier consisting of 3 fully connected layers with dropout.
We are using a learning rate of 6e-4,
Adam [37] as optimizer and train each model for 5 epochs.
At inference time, the post-softmax class distributions [5] of all constituent models are summed up and then evaluated.
Compared to simple majority voting, this method allows for a more fine-grained aggregation of model predictions in the ensembling step.
The authors of the original SISA publication [5] found that this aggregation strategy yields better performance on Imagenet and Mini-Imagenet, while not hurting the performance on the SVHN and Purchase dataset.
During evaluation, we record prediction error rates (1 − accuracy) per class using the balanced test set of 40, samples.
We do not care about the error rates for individual digits, but rather for the error rates depending on if that class was a majority or minority class.
Hence, results are not reported for 10 classes, but rather grouped by class size in the training data or imbalance ratio respectively.
Given our dataset, which was chosen to be as large as possible, we chose 5 shards and 3 slices as a compromise between speedup and sample representation.
If too many slices have 0 samples of a given class, we are not able to draw meaningful conclusions about the methods we are evaluating.
Specifically, the likelihood of having at least one slice with 0 samples of the 1:1000 class using random allocation and having only 24 samples at hand is already 98.5%.
Because of this, we deviated from the default shard and slice values from the original SISA paper [5] but rather addressed their influence in our ablation experiments (Sec. III-E).
We repeat our experiment for a monolithic baseline model and a SISA model.
Both models are trained regularly as well as using all methods against class imbalance described in Sec.
III-B.
For the experiments that use focal loss, we have chosen γ = 1 . 0 .
RUS is evaluated on its own as well as in combination with cost-sensitive learning with regular cross-entropy loss.
The resulting RUS dataset has a size of 170 , 664 / √ 5 = 76 , 323 samples and includes all of the original minority class samples.
It will be trained like a SISA model with 1 shard and 3 slices.
D. Results
Fig. 2 and Tab.
II show the error rates grouped by model type and class imbalance.
The error rate becomes higher when the class imbalance becomes more extreme.
For the monolithic baseline model, mean error rates compared to the majority class are 2 .
7 × as high for an imbalance of 1:10, 14 . 4 × as high for an imbalance of 1:100, and 196 .
3 × as high for an imbalance of 1:1000.
Similar observations can be made for SISA, but the rate of deterioration is considerably higher.
While the majority class performance remains practically unchanged, the mean error rates compared to the majority class are already 5 .
1 × higher for an imbalance of 1:10, 40 .
5 × higher for an imbalance of 1:100, and 242 .
8 × higher for an imbalance of 1:1000.
Compared to the previous rate of deterioration, this reflects an increase of +92%, +184%, and +24.5% respectively.
One should note that the performance deterioration for the classes imbalanced by 1:1000 has already reached its natural maximum (100% error rate/0% accuracy), and thus also the +24.5% change in the speed of deterioration reflects the maximum possible rate of increase.
These results suggest that not only does the performance of minority classes also deteriorate when using SISA, but it does so much more quickly than if SISA were not used for training.
Almost all evaluated methods against class imbalance improve the performance of minority classes both in the monolithic as well as the SISA model.
Cost-sensitive learning with
LDAM loss improves the performance of the most extreme 1:1000 classes but yields higher error rates for the majority and some of the other minority classes.
In the monolithic model, cost-sensitive learning with regular cross-entropy loss delivers the best overall error rates.
In the SISA model, cost-sensitive learning with focal loss has the best overall error rates.
However, two observations hold for all evaluated methods:
First, that the minority class error rates become larger multiples of the majority class error rate with higher imbalance ratios, meaning that none of the methods were able to remove the effects of the class imbalance completely.
Second, and more importantly, that the majority class error rate multiples in the SISA models are always larger than the corresponding values in the monolithic model.
For example, while the error rate for the 1:100 class in the monolithic ROS model was 11 .
1 times as high as the majority class error rate, it is 22 .times as high in the SISA model.
This means that while the overall performance of minority classes improves using methods against class imbalance, the unequally distributed burden introduced by SISA remains.
RUS results in majority class error rates worse than both the monolith and the SISA model.
This is not surprising, as it has fewer training examples to learn from.
Combined with costsensitive learning with cross-entropy it outperforms all other evaluated models in the 1:100 and 1:1000 classes and only performs slightly worse than the – slower – monolith model with costs on the 1:10 classes.
The pattern of an increased burden shouldered by minority classes is also reversed – in all cases, the mcer multiple is lower than in the corresponding monolith model.
This improvement was however also paid for with an increased absolute error rate for the majority class.
E. Effect of the Number of Shards and Slices
The experiments conducted in the original SISA publication indicated that the number of shards and slices should be carefully chosen in order to ensure that the accuracy gap between monolithic and SISA model does not become too large.
In a small ablation study, we evaluate how the impact on minority classes changes when the number of shards or the number of slices is increased while keeping the other fixed.
We repeat the same experiment as before with 5, 10, andshards and 3 slices, as well as 5 shards and 3, 6, and 12 slices.
The results for varying shard numbers are shown in Fig.
3 and
Tab. III, the results for varying slice numbers in Fig.
4 and
The results for the number of shards show a relationship between an increasing number of shards and a more pronounced performance gap between majority and minority classes.
Unless the error rate ceiling is reached, the rate of deterioration increases with the number of shards, except for the SISA model with 20 shards and the imbalance ratio of 1:10, which remains almost unchanged.
There seems to be no clear relationship between the number of slices and the performance of minority classes.
These findings align with the results of the original SISA paper [5], where accuracy was also mainly dependent on the number of shards, but less so on the number of slices if the model is trained for enough epochs.
IV. K NOWLEDGE OF P RIVACY R IGHTS I S N OT E VENLY D ISTRIBUTED
As mentioned in the introduction several legislations give people the right to request the deletion of their personal data.
However, who is aware of and exerting this right is not evenly distributed among the population, as we will discuss in this section.
As long as a service provider is able to estimate the likelihood of a given individual to submit a data deletion request with reasonable accuracy, sorting the training samples according to that likelihood will lead to an improvement in the average unlearning speedup by limiting the retraining to fewer shards and slices.
Such estimates can be derived on a per-country level, as simulated by the original authors [5].
Their experiment was motivated by work published by Google [38], which showed differences of up to 1:10 in the number of
URL deletion requests per capita for individual EU member states.
But unlearning likelihood estimates can stem from any number of useful predictive features at the service provider’s [Figure] [Figure] [Table]
[Table]
[Table] [Table] disposal. As many service providers have privacy settings that the user can change, the fact that the user changed those from the default settings may indicate an increased user concern about privacy.
But also other general features such as age, socioeconomic status, gender, education, internet usage, the date of the last login, etc. may serve as useful predictors for the likelihood of an approaching deletion request.
Large search engines and online social networks that earn money through targeted advertising have access to these features.
A. Unlearning for the Young and Rich
After the GDPR came into effect in 2018, the European
Commission Directorate-General for Justice and Consumers commissioned a Eurobarometer survey [39] that set to explore the awareness of the newly introduced legislation in the EU, including questions on data sharing and data protection in general as well as knowledge of the newly introduced rights.
Likewise, Consumer Action and the Consumer Federation of
America , two US non-profits, conducted a survey [40] that evaluated the awareness of and experience with the CCPA among California residents. Both surveys found considerable [Table] [Figure]
[Table] [Table] [Table] dependencies of awareness and exertion of deletion rights and socioeconomic status, age, education, previous privacy setting changes, internet usage, race, gender, and more.
In the EU survey [39], the highest awareness of deletion rights recorded across all categories was among the subgroup of managers, with an awareness rate of 79%.
The EU survey validates privacy setting changes as useful predictor for deletion right awareness and highlights disparities among which groups actually change those settings.
Selected results from both studies are shown in Fig.
If there are complex relationships between socioeconomic factors and unlearning likelihood, it is hard to imagine any classification scenario involving personal data in which the target variables are uncorrelated with those factors.
In the medical domain being sick is positively correlated with age (and negatively with household wealth [41]), in the banking context creditworthiness is positively correlated with age and socioeconomic status, and even in the retail domain customer lifetime value shows a positive correlation with household income.
When class label and unlearning likelihood are correlated and an adaptive SISA strategy is used, this can further amplify class imbalances in shards and slices.
In the following ablation study we will evaluate the effect of correlated (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class-correlated unlearning likelihood on model performance. [Figure] [Figure]
[Figure] [Table] B. Experimental Setup
In the original SISA paper [5], the algorithm presented for distribution-aware sharding sorts all samples by their unlearning likelihood and then places them into shards until the expected cumulative probability E ( χ i ) of that shard being unlearned reaches a threshold C .
Then a new shard is created and filled with the remaining samples until it reaches the threshold again, which is repeated until no samples are left.
This procedure accumulates high-likelihood samples into fewer shards that are smaller in size.
While this does not result in a speedup for a single deletion request, as all shards are equally likely, it decreases the expected number of samples to be retrained for multiple batched requests.
In this ablation study, we are however not interested in investigating the impact of different shard sizes , but the impact of different shard compositions that have been introduced through the sorting step.
We will therefore evaluate the methods for distribution-aware sharding and slicing shown in Fig.
In the fewer shards setting, samples are sorted by their unlearning likelihood , then distributed into n (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distributed (cid:58)(cid:58)(cid:58) into (cid:58)(cid:58) S (cid:58) shards according to their likelihood, and finally sorted into m slices according to their likelihood (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) randomly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) assigned (cid:58)(cid:58) to (cid:58)(cid:58) R (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slices.
In the later slices setting, the sorted samples are first assigned their (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) randomly (cid:58)(cid:58)(cid:58)(cid:58) split (cid:58)(cid:58)(cid:58) up (cid:58)(cid:58)(cid:58)(cid:58) into (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shards, (cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) placed (cid:58)(cid:58) in earlier or later shard number, and only afterwards sorted into shards.
In essence, both strategies can be regarded as the transpose of each other.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slices (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) based (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) lower (cid:58)(cid:58)(cid:58) or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) higher (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] Both methods produce shards and slices of equal size but with different expected unlearning probabilities, which in return result in the intended speedup for multiple deletion requests.
The absence of unequal shard and slice sizes allows us to measure the effects of the data composition, and not the shard size or aggregation method.
For the experiments, we are modeling unlearning likelihood as a normally distributed variable with a given mean and standard deviation, where the mean for all minority classes is either one standard deviation higher or one standard deviation lower than the majority class.
This results in a slightly higher or lower concentration of minority-class samples in the respective slices or shards.
SISA models with 5 shards and 3 slices are then trained in the same way as in the previous experiments without any mitigation method described in Sec.
III-B. Like before, we record error rates grouped by imbalance ratio, which this time also reflect the associated higher or lower average unlearning likelihood. [Table]
For (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) second (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoint, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) slice (cid:58)(cid:58) 1 (cid:58)(cid:58) ∪ (cid:58)(cid:58)(cid:58)(cid:58) slice (cid:58)(cid:58) 2, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) so (cid:58)(cid:58)(cid:58)(cid:58) on.
(cid:58)(cid:58)(cid:58) At (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) same (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) time, (cid:58)(cid:58)(cid:58)(cid:58)
the [Table] samples (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58) last (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slice, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58) could (cid:58)(cid:58)(cid:58)(cid:58) put (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) later (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slices (cid:58)(cid:58) at (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) disadvantage. (cid:58)(cid:58)
To (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) determine (cid:58)(cid:58) if (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) case (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) testing (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) later (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slices (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) strategy (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) variant (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slices (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unioned (cid:58)(cid:58) as (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) progresses (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) number (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) epochs (cid:58)(cid:58)(cid:58) per (cid:58)(cid:58)(cid:58)(cid:58) slice (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58) equal (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)
the
[Table]
number
[Table]
of
[Table]
in
[Table]
the
[Table]
monolith.
[Table]
This (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) means (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)
the
[Table]
total
[Table]
number
[Table]
of
[Table]
number (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) times. (cid:58) C. Results The results for the strategy where samples with a high unlearning likelihood are placed into fewer shards can be seen in Fig.
8, the results for the placement in later trained slices in Fig. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
(regular (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA) (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)
[Figure]
[Figure]
(SISA
[Figure] w/o (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unioning). While the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) While (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) adaptive (cid:58) placement seems to have no large (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) small (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) positive (cid:58) effect on the error rates of the majority class, the performance of minority classes is consistently (cid:58)(cid:58)(cid:58)(cid:58) worse (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) fewer (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shards (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) setting. (cid:58)(cid:58)(cid:58)
In (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) later (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slices (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) setting, (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) performance (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] better when belonging to a minority class is associated with a higher-than-average unlearning likelihood . In fact, the performance of minority classesis as good as when
[Table]
their
[Table]
performance.
[Table] In contrast, when minority classes have a lower than average unlearning likelihood their performance is sometimes but not always worse than when no (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) previous [Table] experiments, (cid:58)(cid:58)(cid:58)
we
[Table] analyzed (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] trained (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) datasets (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) contain (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58) very (cid:58)(cid:58)(cid:58) few (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) protected (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroup (cid:58)(cid:58)(cid:58)(cid:58) tend (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) perform (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) worse (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
[Table]
[Table]
while
[Table]
publicly
[Table]
available
[Table]
facial
[Table]
datasets
[Table]
contain
[Table]
mostly
[Table]
white
[Table]
faces
[Table]
[Table]
[Table]
If
[Table]
SISA
[Table]
has
[Table]
a
[Table]
more
[Table]
detrimental
[Table]
make (cid:58)(cid:58) up (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) smaller (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) portion (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.
(cid:58) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Another (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) issue (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) may (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) arise (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correlations (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) between (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroup (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) membership (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unlearning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) likelihood.
(cid:58)(cid:58)(cid:58) As (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) discussed (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) Sec.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) IV-A, (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) likelihood (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unlearning (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) evenly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distributed: (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) upper (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) younger (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) person, (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) likely (cid:58)(cid:58)(cid:58)(cid:58)
they (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) privacy-aware.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
This (cid:58)(cid:58)(cid:58)(cid:58) gives (cid:58)(cid:58)(cid:58)(cid:58) rise (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58) risk (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
may
[Table]
be
[Table]
or than for others
Experimental (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Setup (cid:58)(cid:58) To (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) determine (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) effect (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) population (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroups, (cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58) run (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiments (cid:58)(cid:58) on (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subset (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) UTKFace (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [43] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
{+imbalance (cid:58)(cid:58)(cid:58)(cid:58) ratio (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) between (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) white (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) black (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroup (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58) 1:
+}
(cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) there (cid:58) is (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correlation (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) between (cid:58)(cid:58)(cid:58)(cid:58) race (cid:58)(cid:58)(cid:58) and
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) label.
(cid:58)(cid:58)(cid:58)(cid:58)
Our
[Table]
ResNet-18 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[36] (cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) makes (cid:58)(cid:58)(cid:58) use (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) techniques (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) improved (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generalization (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) including (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dropout (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) label (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) smoothing.
(cid:58) (cid:58)(cid:58)(cid:58)
We
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) evaluate (cid:58)(cid:58)(cid:58)(cid:58)
[Table]
to
[Table]
when (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) protected (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attribute (cid:58)(cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) completely (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) uncorrelated (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) membership (cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) underrepresented, (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) when (cid:58)(cid:58)
it (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) equivalent (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) membership.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiment (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) repeated (cid:58)(cid:58) 5 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) times.
(cid:58)(cid:58)(cid:58) All (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) models (cid:58)(cid:58)(cid:58)(cid:58) have (cid:58)(cid:58) 5 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shards (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58) 3 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slices (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each
(cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) results (cid:58)(cid:58)(cid:58)(cid:58) can (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seen (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)
The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) average (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) error (cid:58)(cid:58)(cid:58) rate (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) black (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) faces (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) considerably (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) higher (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)
[Table]
error (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) rates (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) young (cid:58)(cid:58)(cid:58)(cid:58) faces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) across (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) runs, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) despite (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classes (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) being (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) balanced, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) likely (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) represents (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) global (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) optimum (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) given (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) binary (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) cross-entropy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) loss.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Training (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classifier
(cid:58)(cid:58) as (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) model (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increased (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
[Figure]
[Table]
and unlearning likelihood exists
[Table]
It is
[Table]
both (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classes (cid:58)(cid:58)(cid:58)(cid:58) still (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) being (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) balanced.
(cid:58)(cid:58)(cid:58)(cid:58)
The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) direction (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) change (cid:58)(cid:58)(cid:58) is (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) line (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) results (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) previous (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiments (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] VI. D ISCUSSION
The results of our first experiment show that the performance gap that comes with using SISA is larger for minority classes than for majority classes.
If we assume that this generalizes to other tasks, this introduces a dilemma:
Is it more important to allow efficient unlearning or is it more important to have a high accuracy across all classes?
A. The Limits of SISA
The original SISA paper [5] highlighted the importance of the absolute number of samples per shard to achieving good generalization and model performance.
We interpret our results as further evidence for this finding but extend the scope to the absolute numbers of samples per shard and class .
The inability of both data-level as well as algorithm-level methods (see Sec.
III-B) to alleviate the problem of increased deterioration rates for minority classes highlights how difficult it is to overcome the increased accuracy gap that SISA introduces.
The fact that the deterioration rate in our experiments was closely related to the number of shards but not the number of slices suggests that the root cause of lower performance for SISA in general and the reason for the disadvantage for minority classes are the same.
Hence, finding solutions for the problems of imbalanced classes with SISA may help improve the accuracy of SISA models in general, even when classes are balanced.
In an apples-to-apples (in terms of retraining time) comparison between SISA and the N/ √ n (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 / √ S (cid:58) RUS baseline, the baseline outperformed SISA on all minority classes.
The superiority of the baseline becomes more pronounced as the imbalance ratio rises.
This means that as long as a slight decrease in majority class performance is acceptable, the performance of minority classes can be boosted considerably by using the baseline while preserving the same averagecase retraining speedup (cid:58)(cid:58)(cid:58) for [Table]
requests). At the same time, using the baseline reduces the worst-case retraining time by a factor of √ n [Table] √ S , and reduces the best-case retraining time to 0.
The lack of an ensembling step comes with further advantages: different learning tasks such as contrastive representation learning, for which ensembling methods are less intuitive than simply summing class probabilities, are now possible too.
Unfortunately, the experiments in the original SISA paper were only conducted in an apple-to-oranges way, where SISA performance was compared to a N/n [Table] 1 /S
[Table] baseline with a different retraining time that puts the baseline at an artificial disadvantage. We leave the question of whether there are even more scenarios in which SISA is outperformed by a N/ √ n baseline to future research [Table]
shards (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Mini-Imagenet (cid:58)(cid:58)(cid:58)(cid:58) had (cid:58)(cid:58) an (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accuracy (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) approximately (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 55%, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) while (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58) 1 /S (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) baseline (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) model (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58) 1 / 5 (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58) data (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (slightly (cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 / √ 20 ) (cid:58)(cid:58)(cid:58)(cid:58) had (cid:58)(cid:58) an (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accuracy (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) approximately (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 61% (cid:58)(cid:58)(cid:58)
[5] (cid:58) . (cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58) same (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observation (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) holds (cid:58)(cid:58)(cid:58)(cid:58) true (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accuracies (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Imagenet (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58) well (cid:58)(cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SVHN, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MNIST, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Purchase (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) when (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) comparing (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (S=50) (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 / 10 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) baseline (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (containing (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly (cid:58)(cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58) data (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 / √ 50 ) (cid:58)(cid:58)(cid:58)(cid:58) [5] (cid:58) . Another observation that can be made (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiments is that the variance of majority class error rates seems to be increase when using RUS.
A possible reason for this could be that important prototypical training samples were randomly removed from the majority classes.
If the majority classes were not down-sampled randomly but strategically (cid:58)(cid:58)(cid:58) (e.g. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58) data (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) pruning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [44] (cid:58) ) (cid:58) the average majority class performance could possibly be improved while preserving the beneficial speedup and class balancing effect.
While the performance of SISA on imbalanced datasets could probably be improved incrementally by combining more algorithm-level methods or even by coming up with more elaborate ensembling methods, the more sensible option is likely choosing the N/ √ n (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 / √ S (cid:58) RUS baseline instead.
In fact, even a N/n (cid:58)(cid:58)(cid:58) 1 /S (cid:58) RUS baseline, which is n (cid:58) S (cid:58) times faster than regular SISA, compared favorably to the SISA minority class error rates when we evaluated it.
The answer to whether SISA or another baseline yields better results depends therefore greatly on the dataset and is not as clear as the original SISA paper [5] likes to make it out.
We were able to show that distribution-aware SISA models are in fact sensitive to correlations between class membership and unlearning likelihood.
When minority class membership is positively (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) negatively correlated with a high unlearning likelihood samples tend to be remembered better by the constituent models.
The reason for this may be that “expert shards” are able to produce logits high enough to skew the aggregated class distribution in the correct direction, or that the placement of samples in later slices increases the ability of the model to remember samples from minority classes, which were seen more recently. [Table] is [Table]
earlier
[Table] are [Table] more
[Table]
more
[Table]
to
[Table]
or
[Table]
the
[Table]
of
[Table]
the
[Table]
However, the fact that even in the fewer shards setting higher unlearning likelihood samples were placed in later slices inside that shard makes it more difficult to distinguish the effects of placement in fewer shards and later slices. Even though less optimal in terms of retraining time, a setting in which samples are sorted in fewer shards but then randomly placed in slices [Table] and a setting in which samples are sorted in later slices but randomly allocated to shards could allow for better experimental isolation of both effects.
In essence [Table] decreasing the density of minority class samples in earlier slices of the model and increasing their density in later slices is equivalent to [Table]
The
[Table]
when
[Table]
reversed
[Table]
effect
[Table]
is
[Table]
likely
[Table]
caused
[Table]
by
[Table]
the
[Table]
model
[Table]
better
[Table]
remembering
[Table]
samples (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58) it (cid:58)(cid:58)(cid:58) was
[Table]
[Table]
on
[Table]
most
[Table]
recently.
(cid:58)(cid:58)(cid:58)(cid:58)
This (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] assigning lower class costs to minority classes at the beginning of the training and assigning higher class costs at the end of the training of the constituent models. This involuntary introduction of cost-sensitivity to the model may also explain the improved performance of minority classes in this setting
[Table] (cid:58) . Parallels can be drawn to dynamic curriculum learning [45], where a scheduler dynamically adapts the label distribution from imbalanced to balanced over time.
Applying dynamic curriculum learning could possibly also further improve the performance of our RUS baseline. [Table] In [Table]
[Table]
gives
[Table] another (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slight (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) speedup (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) retraining (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) time, (cid:58)(cid:58)(cid:58) as (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) relationship (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) between (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) number (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) slices (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) being (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) retrained (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] only (cid:58)(cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) direction (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) opposite (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) effects (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) variants (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) allow (cid:58)(cid:58) us (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) always (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) select (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) whichever (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) variant (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) benefits (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] B. A Potential Attack Vector In the context of distribution-aware SISA unlearning we also want to highlight a potential attack vector that has so far not been mentioned in the literature.
While Marchant, Rubinstein and Alfeld [46] presented an attack on approximate machine unlearning methods that strategically places poisoned samples to incur the maximum possible retraining cost if their deletion is requested, the same might be possible for SISA.
If an attacker is able to produce data points that the service provider will likely consider high or low unlearning likelihood samples, for example by creating user accounts in particular countries or with an appropriate age, such data points will be placed in specific locations in a distribution-aware SISA model.
While this could obviously be used to maliciously increase retraining cost, the predictable placement of data points could also allow for more effective poisoning attacks on the model.
Parallels of this attack vector can be drawn to data ordering attacks [47], where an adversary is able to skew the model towards a desired direction or prevent it from learning altogether purely by manipulating the order in which samples are presented to a machine learning model during training.
Even though our adversary would need to be able to insert novel data points as well, they could also exert some control over when those data points are seen by the model.
Especially when classes are imbalanced and the absolute number of minority class samples is small, changes to the correlation of class label and unlearning likelihood can also be introduced easily by an adversary, and our experiments have shown that such correlations do have an effect on model performance.
As we have not conducted experiments testing the effectiveness of the mentioned attack we leave the exploration of this attack vector to future research.
C. Potential Impact on Fairness
One important observation in our experiments is that classes with higher unlearning likelihood seem to have better accuracy.
As discussed in Sec.
IV-A, the likelihood of unlearning is not evenly distributed: the more upper class and the younger the person, the more likely they are privacy-aware.
This gives rise to the risk of unfairness being introduced in a model by applying a biased function to predict unlearning likelihood .
If privileged population groups are given preferential placement in shards and slices, prediction accuracies for those groups may be higher than for others.
A reverse effect could be observed if our N/ √ n baseline model is combined with an adaptive under-sampling strategy that discards samples with a high unlearning likelihood.
If those samples never make it into the model, populations with a higher unlearning likelihood may have lower accuracy.
Hence, the problem of unlearning likelihood and other parameters being correlated may not only be relevant for regular SISA training. In this study, we analyzed the impact
C. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Subgroups (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Fairness (cid:58)(cid:58)(cid:58)
The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiments (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Section (cid:58)(cid:58)(cid:58) V (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) showed (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increased (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) burden (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) minority (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroups (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) without (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correlation, (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) random (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) adaptive (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) placement (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples.
(cid:58)(cid:58)
In (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) contrast, (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) effects (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unlearning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) likelihood (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correlation (cid:58)(cid:58)(cid:58)(cid:58) with [Table] speaking [Table]
the
[Table]
given
[Table]
drops
[Table]
as
[Table]
the
[Table]
of
[Table]
increases
[Table]
and
[Table]
follows
[Table]
law
[Table]
If
[Table]
is [Table] burden of SISA on imbalanced classes , but an impact on the subgroup fairness of a model is also conceivable. For example, models trained on datasets that contain only very few samples from a protected subgroup tend to perform worse on that subgroup.
If SISA has a more detrimental impact on minority classes it could also be that it hurts the performance of minority subgroupsmore [Table] rate [Table]
as [Table] the [Table]
with
[Table] gets [Table] lower – independent of class membership.
As we have not tested this hypothesis in our own experiments, we suggest that the impact of SISA on fairness should be investigated by future research (cid:58)(cid:58)(cid:58)
the
[Table]
relationship
[Table]
between
[Table]
numbers
[Table]
and
[Table]
black subgroup than the white
At the same time the sensitivity (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) age (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroups (cid:58)(cid:58) to (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) adaptive (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) placement (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) confirms (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) results (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Section (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) IV-A (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shows (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
A concept similar to cost-sensitive learning can be found in fair machine learning [48], where reweighting [49] aims to remove an existing correlation between belonging to a certain subgroup and class membership by assigning associated weights to each group.
Another possible direction for future research could therefore be whether or not the efficacy of fairness-enhancing techniques such as reweighting are impacted by SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Unfairness (cid:58)(cid:58)(cid:58)(cid:58)
can (cid:58)(cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) introduced (cid:58)(cid:58)(cid:58)(cid:58) into (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) models (cid:58)(cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) protected (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroup (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) comprising (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) minority (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Many (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) datasets (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) produce (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) unfair (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classifiers (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) because (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subgroup (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) membership (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correlated (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) positive (cid:58)(cid:58)(cid:58) or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) negative (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] metrics as well. D. The Role of the Dataset
The larger a datasetis (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset, the larger the costs for retraining are and the higher the importance of efficient methods for machine unlearning.
Unfortunately, the EMNIST [30] dataset we used for our experiments can only be considered to be medium-sized [Table] and [Table]
down
[Table] UTKFace [Table] hours, no service provider would be willing to accept the lower performance of SISA in return for saving mere minutes of retraining time.
Since large datasets might provide additional benefits for learning (a 1:1000 minority class in a 10,000,000 sample dataset still consists of 10,000 samples) the trade-off that service providers operating on a global scale face could be more favorable than suggested by our experiments.
One of the goals of this study was to analyze the impact of minority classes without the introduction of additional biases through the dataset.
We were able to demonstrate the effects on minority classes using a synthetically unbalanced dataset without additional confounds.
Whenever datasets with naturally occurring imbalances are used for training additional confounds may exist, and special care should be taken to determine which effect is responsible for reduced model performance.
VII. C ONCLUSION
SISA [5] (Sharded, Isolated, Sliced, and Aggregated) training is a framework that allows efficient machine unlearning.
In this paper, we analyzed the impacts of SISA on the performance of imbalanced classes.
We were able to demonstrate that the difference in error rates between majority and minority classes increased with SISA, even for small imbalance ratios of 1:10.
The increase in performance difference between the majority and minority classes mainly depends on the number of shards of the model.
Both data-level and algorithm-level methods for learning with class imbalance improved minority class accuracy.
Yet, the problem of unequal degradation rates persists when applying those methods.
When the performance of minority classes is important, simply down-sampling the dataset into a more balanced single shard of size N/ √ n (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 / √ S (cid:58) yields much better results than applying SISA while preserving the same average retraining speedup.
We were able to show that SISA does not always win over a smaller model without ensembling and that the makeup of the dataset should play an important role when deciding on which machine unlearning method to choose.
In addition, we were able to demonstrate that SISA models are sensitive to correlations between class membership and unlearning likelihood.
While this relationship can be beneficial to the performance of minority classes, the opposite can be true as well.
We point out a potential attack vector that this relationship could open up to adversaries that aim to reduce the model performance. (cid:58)(cid:58)
As (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) long (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) minority (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classes (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correlated (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) direction, (cid:58)
a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) suitable (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) strategy (cid:58)(cid:58)(cid:58)(cid:58) can (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) selected (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) improves (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) performance.
(cid:58) (cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increased (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) burden (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) SISA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seems (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) depend (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) largely (cid:58)(cid:58) on (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] Our work illustrates the importance of researching the side effects that are associated with machine unlearning and highlights the need for a detailed observation of model performance that goes beyond measuring average accuracy
A CKNOWLEDGMENT
We would like to thank Consumer Action and the Consumer
Federation of America for providing a detailed breakdown of survey responses by groups.
We would further like to thank the reviewers for taking the time to review this manuscript and for their thoughtful comments and efforts toward improving our paper