Autoregressive Perturbations for Data Poisoning
Anonymous Author(s)
AffiliationAddress email
Abstract
The prevalence of data scraping from social media as a means to obtain datasets hasled to growing concerns regarding unauthorized use of data.
Data poisoning attackshave been proposed as a bulwark against scraping, as they make data “unlearnable”by adding small, imperceptible perturbations.
Unfortunately, existing methodsrequire knowledge of both the target architecture and the complete dataset so that asurrogate network can be trained, the parameters of which are used to generate theattack.
In this work, we introduce autoregressive (AR) poisoning, a method thatcan generate poisoned data without access to the broader dataset.
The proposed
AR perturbations are generic, can be applied across different datasets, and canpoison different architectures.
Compared to existing unlearnable methods, our ARpoisons are more resistant against common defenses such as adversarial trainingand strong data augmentations.
Our analysis further provides insight into whatmakes an effective data poison.
Introduction
Increasingly large datasets are being used to train state-of-the-art neural networks [24, 26, 25].
Butcollecting enormous datasets through web scraping makes it intractable for a human to reviewsamples in a meaningful way or to obtain consent from relevant parties [3].
In fact, companies havealready trained commercial facial recognition systems using personal data collected from mediaplatforms [15].
To prevent the further exploitation of online data for unauthorized or illegal purposes,imperceptible, adversarial modifications to images can be crafted to cause erroneous output for aneural network trained on the modified data [12].
This crafting of malicious perturbations for thepurpose of interfering with model training is known as data poisoning.
There are a number of data poisoning approaches with different objectives: backdoor attacks ensurethat pre-specified input features cause inaccurate output [5, 13], feature-collision attacks cause aparticular target example to be misclassified as a base class [28, 42] and indiscriminate poisoningattacks aim to degrade overall test performance [1, 2, 39, 18, 9, 10]. In this work, we focus onindiscriminate poisoning, where the poisoner tries to induce poor performance for the victim networkon the clean distribution.
Typically, the poisoner is allowed to perturb the entire dataset, or a largeportion of it, to prevent others from using it.
Throughout this work, unless stated otherwise, poisoningrefers to the perturbing of every image in the training dataset.
We introduce autoregressive (AR) data poisoning for degrading overall performance of neuralnetworks on clean data.
The perturbations that we additively apply to clean data are generated by
AR processes that are data and architecture-independent.
An AR( p ) process is a Markov chain,where each new element is a linear combination of p previous ones, plus noise.
This means ARperturbations are cheap to generate, not requiring any optimization or backpropagation throughnetwork parameters.
AR perturbations are generic; the same set of AR processes can be re-used togenerate diverse perturbations for different image sizes and new datasets, unlike other poisoningmethods which need to train a surrogate network on the target dataset before crafting perturbations.
Our method also provides new insight into why data poisoning works.
We work on top of the result thateffective poisons are typically easy to learn [27] and construct AR perturbations which are separableby a manually-specified CNN.
Working under the intuition that highly separable perturbations shouldbe easily learned, we use the manual specification of parameters as a way of demonstrating that
our
AR perturbations are easily separable.
Our manually-specified CNN makes use of what we call ARfilters, which are attuned to detect noise from a specific AR process.
AR poisoning’s effectiveness iscompetitive or better than error-maximizing, error-minimizing, and random noise poisoning across arange of architectures, datasets, and common defenses.
AR poisoning represents a paradigm shiftfor what a successful indiscriminate poisoning attack looks like, and raises the question of whetherstrong indiscriminate poisons need to be generated by surrogate networks for a given dataset.
Background & Related Work
Error-minimizing and Error-maximizing Noise.
To conduct poisoning attacks on neural networks,recent works have modified data to explicitly cause gradient vanishing [30] or to minimize the losswith respect to the input image [18].
Images perturbed with error-minimizing noises are a surprisinglygood data poisoning attack.
A ResNet-18 (RN-18) trained on a CIFAR-10 [20] sample-wise errorminimizing poison achieves 19 .
9% final test accuracy, while the class-wise variant achieves 16 .
4%final test accuracy after 60 epochs of training [18].
More recently, strong adversarial attacks, whichperturb clean data by maximizing the loss with respect to the input image, have been shown to be themost successful approach thus far [10].
An error-maximizing poison can poison a network to achieve6 .
25% test accuracy on CIFAR-10.
But both error-minimizing and error-maximizing poisons requirea surrogate network, from which perturbations are optimized.
The optimization can be expensive.
Forexample, crafting the main CIFAR-10 poison from [10] takes roughly 6 hours on 4 GPUs.
In contrast,our AR perturbations do not require access to network parameters and can be generated quickly,without the need for backpropagation or a GPU.
We provide a technical overview of error-minimizingand error-maximizing perturbations in Section 3.1.
Random Noise.
Given their simplicity, random noises for data poisoning have been explored asnecessary baselines for indiscriminate poisoning.
If random noise, constrained by an ℓ ∞ norm, isapplied sample-wise to every image in CIFAR-10, a RN-18 trained on this poison can still generalizeto the test set, with ~ 90% accuracy [10, 18].
But if the noise is applied class-wise, where everyimage of a class is perturbed by the same noise, then a RN-18 trained on this CIFAR-10 poison willachieve around chance accuracy; i .
e .
~ 10% [38, 18, 27].
The random perturbations of [38] consistof a fixed number of uniform patch regions, and are nearly identical to the class-wise poison, called“Regions-16,” from [27].
All the random noises that we consider are class-wise, and we confirm theywork well in a standard training setup using a RN-18, but their performance varies across architecturesand they are rendered ineffective against strong data augmentations like Cutout [7], CutMix [40], and
Mixup [41].
Conversely, our AR poisons degrade test performance more than error-maximizing, errorminimizing, and random poisons on almost every architecture.
We show that AR perturbations areeffective against strong data augmentations and can even mitigate some effects of adversarial training.
Understanding Poisoning.
A few works have explored properties that make for effective poisons.
Forexample, [27] find that poisons which are learned quickly have a more harmful effect on the poisontrained network, suggesting that the more quickly perturbations help minimize the training loss, the more effective the poison is.
[38] perform a related experiment where they use a single linear layer,train on perturbations from a variety of poisons, and demonstrate that they can discriminate the classof the error-minimizing and error-maximizing noises with high accuracy.
Their result indicates thatpoisoning works by using perturbations that are almost linearly separable.
We make use of ideas fromboth papers, designing AR perturbations which are provably separable and Markovian in local regions.
Other Related Works.
Several works have also focused on variants of “unlearnable” poisoningattacks.
[9] propose to employ gradient alignment [11] to generate poisons.
But their method iscomputationally expensive; it requires a surrogate model to solve a bi-level objective.
[39] proposegeneration of an unlearnable dataset using neural tangent kernels.
Their method also requires traininga surrogate model, takes a long time to generate, and does not scale easily to large datasets.
Incontrast, our approach is simple and does not require surrogate models.
[23] propose an invertibletransformation to control learnability of a dataset for authorized users, while ensuring the dataremains unlearnable for other users.
[34] showed that data poisoning methods can be broken usingadversarial training.
[29] and [36] propose variants of error-minimizing noise to defend againstadversarial training.
Our AR poisons do not focus on adversarial training.
While adversarial trainingremains a strong defense, our AR poisons show competitive performance.
We discuss adversarialtraining in detail in Section 4.3.2.
A thorough overview of data poisoning methods, including thosethat do not perturb the entire training dataset, can be found in [12].
Autoregressive Noises for Poisoning
3.1 Problem Statement
We formulate the problem of creating a clean-label poison in the context of image classification with
DNNs, following [18].
For a K -class classification task, we denote the clean training and test datasetsas D c and D t , respectively.
We assume D c , D t ∼ D .
We let f θ represent a classification DNN withparameters θ .
The goal is to perturb D c into a poisoned set D p such that when DNNs are trained on
D p , they perform poorly on test set D t .
Suppose there are n samples in the clean training set, i.e. D c = { ( x i , y i ) } ni =1 where x i ∈ R d arethe inputs and y i ∈ { 1 , ..., K } are the labels.
We denote the poisoned dataset as D p = { ( x ′ i , y i ) } ni =1where x ′ i
=
x i + δ
i is the poisoned version of the example x i ∈ D c and where δ i ∈ ∆ ⊂ R d is theperturbation.
The set of allowable perturbations, ∆ , is usually defined by ∥ δ ∥ p < ϵ where ∥ · ∥ p is theℓ p norm and ϵ is set to be small enough that it does not affect the utility of the example.
In this work,we use the ℓ 2 norm to constrain the size of our perturbations for reasons we describe in Section 3.3.1.
Poisons are created by applying a perturbation to a clean image in either a class-wise or sample-wisemanner.
When a perturbation is applied class-wise, every sample of a given class is perturbed inthe same way.
That is, x ′ i
= x i + δ y
i
and δ y i ∈
∆ C = {
δ 1 , ..., δ K } .
Due to the explicit correlationbetween the perturbation and the true label, it should not be surprising that class-wise poisonsappear to trick the model to learn the perturbation over the image content, subsequently reducinggeneralization to the clean test set.
When a poison is applied sample-wise, every sample of the trainingset is perturbed independently.
That is, x ′ i
=
x i + δ i and δ i ∈ ∆ S
= { δ 1 , ..., δ n } .
Because class-wiseperturbations can be recovered by taking the average image of a class, these should therefore beeasy to remove.
Hence, we focus our study on sample-wise instead of class-wise poisons.
We stillcompare to simple, randomly generated class-wise noises shown by [18] to further demonstrate theeffectiveness of our method.
Error-maximizing poisoning [10] aims to solve the following objective in terms of perturbationsδ
i ∈ ∆ to samples x
i ∈ D c :
[Equation]
) whereas error-minimizing poisons [18] solve the following objective:
[Equation]
Both objectives are bi-level optimization problems, as θ is implicitly defined via:
[Equation]
In both error-maximizing and error-minimizing poisoning, the adversary intends for a network, f ,trained on the poison to perform poorly on the test distribution D t , from which D c was also sampled.
But the way in which both methods achieve the same goal is distinct.
3.2 Generating Autoregressive Noise
Autoregressive (AR) perturbations have a particularly useful structure where local regions throughoutthe perturbation are Markovian, exposing a linear dependence on neighboring pixels [37].
Thisproperty is critical as it allows for a particular filter to perfectly detect noise from a specific ARprocess, indicating the noise is simple and potentially easily learned.
We develop a sample-wise poison where clean images are perturbed using additive noise.
For each x iin the clean training dataset, our algorithm crafts a δ i , where ∥ δ i ∥ 2 = ϵ , so that the resulting poisonimage is x ′ i
= x i + δ i .
The novelty of our method is in how we find and use autoregressive (AR)processes to generate δ i .
In the following, let x t refer to the t th entry within a sliding window of δ i .
An autoregressive (AR) process models the conditional mean of x t , as a function of past observationsx t − 1 , x t − 2 , ..., x t − p in the following way:
[Equation]
where ϵ t is an uncorrelated process with mean zero and β i are the AR process coefficients.
Forsimplicity, we set ϵ t = 0 in our work.
An AR process that depends on p past observations is called an
AR model of degree p , denoted AR( p ).
For any AR( p ) process, we can construct a size p + 1 filterwhere the elements are β p , ..., β 1 and the last entry of the filter is − 1 .
This filter produces a zeroresponse for any noise generated by the AR process with coefficients β p , ..., β 1 .
We refer to this filteras an AR filter, the utility of which is explained in Section 3.3 and Appendix A.1.
Suppose we have a K class classification problem of H × W × C dimensional images.
Additionallysuppose we have a set of K AR processes, A , where each element is itself a set of C AR processes. We first choose the y th i set of C AR processes, A y i .
For each of the C channels, we will be applyingan AR process from A y i inside a V × V sliding window.
Naturally, using an AR process requiresprevious observations, so we populate the perturbation vector δ i with Gaussian noise for the first V − 1 columns and rows.
The V × V sliding window starts at the top left corner of δ i .
Within thissliding window, we apply the AR( V 2 − 1 ) process: the first V 2 − 1 entries in the sliding window areconsidered previous observations, and the V th entry is computed by Eq.
The window is slid left toright, top to bottom until the first channel of δ i is filled.
We then proceed to use the next AR( V 2 − 1 )process in A y
i for the remaining C − 1 channels.
Finally, we crop out the Gaussian noise and scaleδ i to be of size ϵ in ℓ 2 .
Note that this sliding window procedure is identical to that of a convolution.
That is by design, and we explain why it is important in Section 3.3.
A high-level overview of thisalgorithm is illustrated in Figure 2.
Additional details are in Appendix A.3.2.
While we describe our use of AR processes on C -channel images, our method could, in principle, be applied to data otherthan images.
Note that these AR perturbations can be generated independent of the data.
3.3 Why do Autoregressive Noises Work?
Perturbations that are easy to learn have been shown to be more effective at data poisoning [27].
Thus, we seek imperceptible perturbations that are easy to learn.
We propose a simple hypothesis:if there exists a simple CNN that can classify classes of noise perfectly, then these noises will beeasy to learn.
The noises can then be applied to clean images and serve as a shortcut for learning bycommonly-used CNNs.
Autoregressive noises, despite looking visually complex, are actually very simple.
To demonstratetheir separability, we manually specify the parameters of a simple CNN that classifies AR perturbations perfectly.
We begin by designing what we call an AR filter.
Lemma 3.1.
Given an AR perturbation δ , generated from an AR( p ) with coefficients β 1 , ..., β p , thereexists a linear, shift invariant filter where the cross-correlation operator produces a zero response.
We provide a proof in Appendix A.1.
The construction of an AR filter that produces a zero responsefor any noise generated from the corresponding AR process is useful because we can construct a CNNwhich makes use of solely these AR filters to classify noises.
That is, given any AR perturbation,the AR filter with the zero response correctly designates the AR process from which the noise wasgenerated.
In Appendix A.2, we verify this claim by specifying the 3 -layer CNN that can perfectlyclassify AR perturbations.
Crucially, we are not interested in learning classes of noises.
Rather, we are interested in learningclasses of clean data perturbed by noises.
Nevertheless, the characterization of our AR perturbationsas easy to learn, demonstrated by the manual specification of a 3 -layer CNN, is certainly an indicationthat, when applied to clean data, AR perturbations can serve as bait for CNNs.
Our experimentswill seek to answer the following question: If we perturb each sample in the training dataset with animperceptible, yet easily learned AR perturbation, can we induce a learning “shortcut” that minimizesthe training loss but prevents generalization?
3.3.1 Finding AR Process Coefficients
An AR process that employs random coefficients is likely to diverge.
On the other hand, an ARprocess that converges too quickly will mean most values in δ i are close to zero. Due to convergence tovalues near zero, we scale AR perturbations to be exactly of size ϵ in ℓ 2 .
We measure AR perturbationsin ℓ 2 because measuring in ℓ ∞ would underestimate the extent to which these perturbations are lessperceptible than purely ℓ ∞ random noise.
To level the playing field among all poisoning methods,we measure all perturbations using an ℓ 2 norm in this work.
While there exist conditions for AR coefficients for which the process converges [37, 4], we foundthat randomly searching for coefficients was the simplest way to collect a large number of suitable ARprocesses and set a baseline. Put simply, we perform a greedy search where we maximize the responseof one AR filter on the noise generated by a different AR process.
This way, we end up with a set of K AR processes that are sufficiently different.
A benefit of our method is that these AR processescan be reused for poisoning multiple datasets later on.
More details can be found in Appendix A.3.1.
Experiments
We demonstrate the generality of AR poisoning by creating poisons across four datasets, includingdifferent image sizes and number of classes.
Notably, we use the same set of AR processes to poison
SVHN [22], STL-10 [6], and CIFAR-10 [20] since all of these datasets are 10 class classificationproblems.
We demonstrate that despite the victim’s choice of network architecture, AR poisons candegrade a network’s accuracy on clean test data.
We show that while strong data augmentations arean effective defense against all poisons we consider, AR poisoning is largely resistant.
Adversarialtraining and diluting the poison with clean data remain strong defenses, but our AR poisoning methodis competitive with other poisons we consider.
All experiments follow the same general pattern: we train a network on a poisoned dataset and thenevaluate the trained network’s performance on clean test data.
A poison is effective if it can cause thetrained network to have poor test accuracy on clean data, so lower numbers are better throughoutour results.
We say that poisoning effectiveness drops from setup A to setup B if the network frompoison-trained on setup B has higher test set accuracy than the network poison-trained on setup A.
Experimental Settings.
We train a number of ResNet-18 (RN-18) [14] models on different poisonswith cross-entropy loss for 100 epochs using a batch size of 128 .
For our optimizer, we use SGDwith momentum of 0 .
9 and weight decay of 5 ×
10 − 4 .
We use an initial learning rate of 0 . 1 , whichdecays by a factor of 10 on epoch 50 .
In Table 2, we use use the same settings with different networkarchitectures.
4.1 Error-Max, Error-Min, and other Random Noise Poisons
SVHN [22], CIFAR-10, and CIFAR-100 [20] poisons considered in this work contain perturbationsof size ϵ = 1 in ℓ 2 , unless stated otherwise.
For STL-10 [6], all poisons use perturbations of sizeϵ = 3 in ℓ 2 due to the larger size of STL-10 images.
In all cases, perturbations are normalized andscaled to be of size ϵ in ℓ 2 , are additively applied to clean data, and are subsequently clamped to be inimage space.
Dataset details can be found in Appendix A.5.
A sampling of poison images and theircorresponding normalized perturbation can be found in Figure 3 and Appendix A.4.
In our results,class-wise poisons are marked with ◦ and sample-wise poisons are marked with • .
Error-Max and Error-Min Noise.
To generate error-maximizing poisons, we use the open-sourceimplementation of [10].
In particular, we use a 250-step ℓ 2 PGD attack to optimize Eq.
Togenerate error-minimizing poisons, we use the open-source implementation of [18], where a 20-stepℓ 2 PGD attack is used to optimize Eq.
For error-minimizing poisoning, we find that moving in ℓ 2normalized gradient directions is ineffective at reaching the required universal stop error [18], so wemove in signed gradient directions instead (as is done for ℓ ∞ PGD attacks).
Regions-4 and Regions-16 Noise.
Synthetic, random noises are also dataset and network independent.
Thus, to demonstrate the strength of our method, we include three class-wise random noises in ourexperiments.
To generate what we a call a Regions- p noise, we follow [38, 27]: we sample p
RGB vectors of size 3 from a Gaussian distribution and repeat each vector along height and widthdimensions, resulting in a grid-like pattern of p uniform cells or regions.
Assuming a square image ofside length L , a Regions- p noise contains patches of size L √ p × L √ p .
Random Noise.
We also consider a class-wise random noise poison, where perturbations for eachclass are sampled from a Gaussian distribution.
4.2 AR Perturbations are Dataset and Architecture Independent
Unlike error-maximizing and error-minimizing poisons, AR poisons are not dataset-specific.
Onecannot simply take the perturbations from an error-maximizing or error-minimizing poison and applythe same perturbations to images of another dataset.
Perturbations optimized using PGD are knownto be relevant features, necessary for classification [10, 19].
Additionally, for both these methods,a crafting network trained on clean data is needed to produce reasonable gradient information.
Incontrast, AR perturbations are generated from dataset-independent AR processes.
The same set of
[Table] [Table] AR processes can be used to generate the same kinds of noise for images of new datasets.
Buildingfrom this insight, one could potentially collect a large set of K AR processes to perturb any datasetof K or fewer classes, further showing the generality of our method.
In Table 1, we use the same 10 AR processes to generate noise for images of SVHN, STL-10,and CIFAR-10.
AR poisons are, in all cases, either competitive or the most effective poison – apoison-trained RN-18 reaches nearly chance accuracy on STL-10 and CIFAR-10, and being thesecond-best on SVHN and CIFAR-100.
The generality of AR perturbations to different kinds ofdatasets suggests that AR poisoning induces the most easily learned correlation between samples andtheir corresponding label.
We also evaluate the effectiveness of our AR poisons when different architectures are used for training.
Recall that error-maximizing and error-minimizing poisoning use a crafting network to optimizethe input perturbations.
Because it may be possible that these noises are specific to the networkarchitecture, we perform an evaluation of test set accuracy on CIFAR-10 after poison training VGG-19[31], GoogLeNet [32], MobileNet [16], EfficientNet [33], DenseNet [17], and ViT [8].
Our ViT usesa patch size of 4 .
In Table 2, we show that Error-Max and Error-Min poisons generalize relativelywell across a range of related CNNs, but struggle with ViT, which is a transformer architecture.
Incontrast, our AR poison is effective across all CNN architectures and is the most effective poisonagainst ViT. Our AR poison is much more effective over other poisons in almost all cases, achievingimprovements over the next best poison of 4% on RN-18, 5 .
8% on ViT, and 7 . 5% on GoogLeNet.
The design of AR perturbations is meant to target the convolution operation, so it is surprising to seea transformer network be adversely affected.
We believe our AR poison is particularly effective on
GoogLeNet due to the presence of Inception modules that incorporate convolutions using variousfilter sizes.
While our AR perturbations are generated using a 3 × 3 window, the use of various filtersizes may exaggerate their separability, as described in Section 3.3.
4.3 AR Perturbations Against Common Defenses
4.3.1 Data Augmentations and Smaller Perturbations
Our poisoning method relies on imperceptible AR perturbations, so it is conceivable that one couldmodify the data to prevent the learning of these perturbations.
One way of modifying data is by using [Table] • Error- [Table]
[Table]
• Error-Min [18]
[Table]
◦
[Table]
[Table]
◦
[Table]
[Table]
• Autoregressive (Ours)
[Table]
• Error-Max [
[Table]
• Error-Min [18]
[Table] ◦ [Table] [Table]
◦ Regions- [Table] • Autoregressive (Ours)
[Table]
data augmentation strategies during training.
In addition to standard augmentations like random cropsand horizontal flips, we benchmark our AR poison against stronger augmentations like Cutout [7],
CutMix [40], and Mixup [41] in Table 3.
Generally, Mixup seems to be the most effective at disablingpoisons.
A RN-18 poison-trained using standard augmentations plus Mixup can achieve a boosts intest set performance of 13 .
68% on Error-Max, 16 .
42% on Error-Min, 19 . 85% on Regions-4, 5 .
05%on Regions-16, and 5 .
19% on Random Noise.
However, a RN-18 poison-trained on our AR poison( ϵ = 1 ) using standard augmentations plus Cutout, CutMix, or Mixup cannot achieve any boost intest set performance.
We also present results for poisons using perturbations of size ϵ = 0 .
5 to explore just how smallperturbations can be made while still maintaining poisoning performance.
Under standard augmentations, going from larger to smaller perturbations ( ϵ = 1 to ϵ = 0 . 5 ), poison effectiveness drops by8 .
2% for Error-Max, 21 . 13% for Error-Min, 36 .
73% for Regions-4, and 31 .
6% for Regions-16.
Our
AR poison achieves the smallest drop in effectiveness: only 2 .
Random noise can no longer beconsidered a poison at ϵ = 0 .
– it completely breaks for small perturbations.
Under all strong dataaugmentation strategies at ϵ = 0 . 5 , AR poisoning dominates.
For example, under Mixup, the bestrunner-up poison is Error-Max with an effectiveness that is more than 23% lower than AR.
Unlike allother poisons, AR poisoning is exceptionally effective for small perturbations.
Note that in all three augmentation strategies pixels are either dropped or scaled.
Our method isunaffected by these augmentation strategies, unlike error-maximizing, error-minimizing, and otherrandomly noise poisons.
Scaling an AR perturbation does not affect how the corresponding matching
AR filter will respond, 1 and thus, the patterns remain highly separable regardless of perturbation size.
Additionally, AR filters contain values which sum to 0, so uniform regions of an image also producea zero response.
4.3.2 Adversarial Training
Adversarial training has also been shown to be an effective counter strategy against ℓ p -norm constrained data poisons [18, 10, 9, 34].
Using adversarial training, a model trained on the poisoned datacan achieve nearly the same performance as training on clean data [29].
However, adversarial trainingis computationally more expensive than standard training and leads to a decrease in test accuracy[21, 35] when the perturbation radius, ρ a , of the adversary is large.
In Table 5, we include adversarialtraining results on clean data to outline this trade-off where training at large ρ a comes at the costof test accuracy.
A recent line of work has therefore focused on developing better data poisoningmethods that are robust against adversarial training [29, 36] at larger adversarial training radius ρ a .
In Table 5, we compare performance of different poisons against adversarial training.
We perform ℓ 2adversarial training with different perturbation radii, ρ a , using a 7-step PGD attack with a step-size of [Table] [Table] ρ a / 4 .
We also show the performance of adversarial training on clean data.
Data poisoning methodsare fragile to adversarial training even when the training radius ρ a is smaller than poisoning radius ϵ[29, 36].
It is desirable for poisons to remain effective for larger ρ a , because the trade-off betweenstandard test accuracy and robust test accuracy would be exaggerated further.
As shown in the Table 5,when the adversarial training radius ρ a increases, the poisons are gradually rendered ineffective.
Allpoisons are nearly ineffective at ρ a = 0 .
Our proposed AR perturbations remain more effective atsmaller radius, i .
e .
ρ a = 0 .
125 and ρ a = 0 .
25 compared to all other poisons.
4.3.3 Mixing Poisons with Clean Data
Consider the scenario when not all the data can be poisoned.
This setup is practical because, to apractitioner coming into control of poisoned data, additional clean data may be available throughother sources.
Therefore, it is common to evaluate poisoning performance using smaller proportionsof randomly selected poison training samples [10, 18, 29].
A poison can be considered effectiveif the addition of poisoned data hurts test accuracy compared to training on only the clean data.
In
Table 4, we evaluate the effectiveness of poisons using different proportions of clean and poisoneddata.
The top row of Table 4 shows test accuracy after training on only the subset of clean data, withno poisoned data present.
Our AR poisons remain effective compared to other poisons even whenclean data is mixed in.
AR poisons are much more effective when a small portion of the data is clean.
For example, when 5% of data is clean, a model achieves ~ 75% accuracy when training on onlythe clean proportion, but using an additional 95% of AR data leads to a ~ 9% decrease in test setgeneralization.
Our results on clean data demonstrate that AR poisoned data is worse than uselessfor training a network, and a practitioner with access to the data would be better off not using it.
Conclusion
Using the intuition that simple noises are easily learned, we proposed the design of AR perturbations,noises that are so simple they can be perfectly classified by a 3 -layer CNN where all parameters aremanually-specified.
We demonstrate that these AR perturbations are immediately useful and makeeffective poisons for the purpose of preventing a network from generalizing to the clean test distribution.
Unlike other effective poisoning techniques that optimize error-maximizing or error-minimizingnoises, AR poisoning does not need access to a broader dataset or surrogate network parameters.
Weare able to use the same set of 10 AR processes to generate imperceptible noises able to degrade thetest performance of networks trained on three different 10 class datasets.
Unlike randomly generatedpoisons, AR poisons are more potent when training using a new network architecture or strongdata augmentations like Cutout, CutMix, and Mixup.
Against defenses like adversarial training, ARpoisoning is competitive or among the best for a range of attack radii.
Finally, we demonstratedthat AR poisoned data is worse than useless when it is mixed with clean data, reducing likelihoodthat a practitioner would want to include AR poisoned data in their training dataset.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix A.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.1(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.3 andAppendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Full algorithm
and details are in Appendix A.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4 and
Appendix A.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/
A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Appendix
A.1 Proof of Lemma 3.
Lemma A.1.
Given an AR perturbation δ , generated from an AR( p ) with coefficients β 1 , ..., β p , thereexists a linear, shift invariant filter where the cross-correlation operator produces a zero response.
Proof.
Consider any size p + 1 window of the AR perturbation δ where, by definition, the last entryfollows Eq.
(4) directly: [ x t − p , x t − ( p − 1) , ..., x t − 1 , x t ] .
We construct a size p + 1 filter where theelements are β p , ..., β 1 and the last entry of the filter is − 1 .
Notice that the dot product of a windowof δ with this filter is:
[Equation]
[Equation] [Equation] (7) = which implies: [Equation]
[Equation]
) where ⋆ denotes the cross-correlation operator.
A.2 Manually-specified CNN with AR filters
[Figure]
Assume AR perturbations are expected to be of size 32 × 32 , as in CIFAR-10.
To illustrate the easewith which a set of AR perturbations can be classified, we construct a set of 10 AR( 8 ) processes
A = ⃗β 1 , ..., ⃗β 10 , where ⃗β i =
[ β i, 1 , ..., β i, 8 ] is a vector of AR( 8 ) coefficients.
Our network consistsof 3 layers: a convolutional layer, a max-pool layer, and a fully connected layer.
We specify theparameters of each layer below:
Convolutional Layer: 10 AR filters of size 3 × 3 .
No bias vector.
ReLU activation on output.
The i th AR filter is meant to produce a zero response on noise generated from AR process with coefficients ⃗β i (See A.1):
[Equation]
Max-Pool layer: Kernel size of 30 × 30 , the same dimensions as the output of the convolutional layer.
Linear layer: We set W = − I and
b = ⃗ 1 , where ⃗ 1 is a vector of all ones.
Softmax activation on output.
To classify AR perturbation from 10 classes, this network must output a probability distribution over10 classes.
This simple CNN works by assigning a high value to the i th logit when the i th filterproduces a zero response.
For example, when an AR filter outputs a zero response, the activationafter ReLU and Max-Pool is still 0 .
The linear layer will only add a bias of 1.
To confirm our network design, we generate 5 , 000 AR perturbations per AR process, and use thenetwork defined above to classify the noises.
The manually-specified network achieves perfectaccuracy without any training .
We use this result as a demonstration that AR perturbations areseparable.
Note that for other simple, linearly separable image datasets it is non-trivial to manuallyspecify a CNN for perfect accuracy.
Again, our motivation in exploring easily classified noises isthere is substantial work suggesting dataset separability is the key to crafting good poisons.
Output
1: A ← {} 2: while | A | ̸ = K do 3: ⃗β ← N (0 , 1) 4: ⃗β ← ⃗β ∥ ⃗β ∥5: δ ← ARGenerate( ⃗β ) 6: if δ is stable then 7:
Let m store the min response so far.
8: for i in 1,..., | A | do 9: f ← ARFilter( A i ) 10: r ← ConvResponse( δ, f ) 11: m ← min( m, r ) 12: end for 13: if m ≥ T then 14:
A ← A ∪ { β i } 15: end if 16: end if 17: end while 18: return A δ end δ i x x
i
: return i
We define important functions and variables of Algorithm 1 and Algorithm 2:
• ⃗β is a vector where the entries are the AR process coefficients β 1 , ..., β p .
• ARGenerate( ⃗β ) takes a set of AR process coefficients and uses the AR process withina sliding window which goes left to right, top to bottom.
The procedure is described inSection 3.2 and illustrated in Figure 2.•
ARFilter( A i ) takes a set of AR process coefficients and returns the corresponding AR filter,where the entries are the AR coefficients followed by a − 1 .
The AR filter is described inSection 3.2 and used in the construction of our simple AR noise classifier in Appendix A.2.• ConvResponse( δ, f ) takes a signal and filter as input.
It measures a filter f ’s “response” onδ by performing 2D convolution, ReLU, and sum.
A.3.1 Additional Details: Finding AR Process Coefficients
We find that the likelihood of a convergent AR process is much higher when the coefficients arenormalized to sum to 1. Normalizing the coefficients is also important because we want AR filters toproduce a zero response to uniform regions of a clean image.
If the AR coefficients sum to 1 and thefinal entry of the AR filter is − 1 , this ensures that the 2D cross-correlation operator produces a zeroresponse if pixels in a local window are approximately equal.
Specifically, we conduct a random search for K sets of converging AR coefficients by randomlysampling values from a Gaussian distribution and ensuring the resulting process is stable; i .
e . doesnot diverge.
We optimize so that every AR process is sufficiently different from other processesalready included in our growing set.
We do this by measuring the response of AR filters from ARprocesses already in our set to noise generated by the currently sampled AR process.
It may bepossible to optimize a different criteria, but in our implementation, ConvResponse( δ, f ) measures afilter’s response by performing 2D convolution, ReLU, and sum.
We fully outline this random searchfor K sets of AR process coefficients in Algorithm 1.
We conduct a search for K = 10 sets of AR coefficients using Algorithm 1 with a threshold T = 10 .
This search took roughly 11 hours running on a single CPU.
For CIFAR-100, we conduct a search for
K = 100 sets of AR coefficients with T = 3 due to the larger number of classes.
This search tookroughly 4 hours running on a single GPU.
Because Algorithm 1 samples coefficients independentlyon every iteration, significant speedups can likely be achieved using parallelized code.
A.3.2 Additional Details: Generating AR Perturbations
In addition to normalizing and scaling AR perturbation after generation, we also find that croppingout the first few columns and rows, where the Gaussian start signal was generated (see Figure 2),leads to more effective AR perturbations.
We tried cropping only the Gaussian start signal, andcropping the start signal plus additional columns and rows.
We found that cropping the start signalplus two extra columns and rows worked best.
The AR perturbations are also less perceptible thisway, given that the Gaussian noise starting signal has been removed.
For 32 × 32 dimensional images,we generate AR perturbations of size 36 × 36 and crop the first 4 rows and columns.
For 96 × 96dimensional images, we generate AR perturbations of size 100 × 100 and crop the first 4 rows andcolumns.
A.4 Samples of CIFAR-10 Poisons
A.5 Dataset Details
SVHN [22] contains 10 classes, where 73257 images are for training and 26032 images are fortesting.
STL-10 [6] conatins 10 classes, where 5000 are for training and 8000 are for testing.
Both
CIFAR-10 and CIFAR-100 [20] contain 50000 images for training and 10000 images for testing.
Images of CIFAR-10 belong to one of 10 , whereas images from CIFAR-100 belong to one of 100classes.
SVHN, CIFAR-10, and CIFAR-100 contain images of size 32 × 32 × 3 ; STL-10 containsimages of size 96 × 96 .
A.6 Computing Resources
To run all of our experiments, we use an internal cluster.
Jobs were run on a maximum of 4 NVIDIA
GTX 1080Ti.
A.7 t-SNE of Poisons
In analyzing the simplicity of AR perturbations, it is also appropriate to better understand other kindsof perturbations.
In Figure 12, we use t-SNE to plot the perturbations of other poisons considered inthis work.
The clustering of error-minimizing poisons is particularly interesting, with the noise beingmore separable than that of AR.
A.8 Limitations
There are a number of modifications which could potentially make for more effective AR perturbations,the exploration of which we leave to future work.
For example, the size of the sliding window or theorder of the AR process could potentially lead to perturbation values which converge more slowly.
Ahigher order AR process could also be applied along the channel dimension.
As described in Section 3.2, poisoning a K -class classification dataset requires K AR processes.
Arandom search for AR coefficients is certainly an area of improvement.
Nevertheless, having found aset of K AR processes, Table 1 is evidence that they can be reused for any other classification datasetwith K or fewer classes.
A.9 Broader Impact Statement
While data poisoning could be used as an attack, the research community has actively been interestedin its use for privacy protection too.
Our work has the potential to protect private data or to allow forthe secure release of datasets.
The likelihood that an adversary is able to perturb an entire dataset forthe purpose of preventing generalization to a test set is miniscule.
On the other hand, the likelihoodthat a private entity might want methods that allow them to release a dataset securely is more likely.
Our work is motivated by understanding the kinds of features that deep neural networks choose tolearn.
A.10 10 effective AR Processes
On the final pages, we provide coefficients for the 10 AR( 8 ) processes which are used to generatethe AR poisons for SVHN, CIFAR-10, and STL-10.
Because we apply a different AR process foreach channel in 3 -channel images, there are really 30 AR( 8 ) processes (each 3 × 3 array defines an
AR( 8 ) process).
Each 3 × 3 × 3 dimensional array defines the three AR processes needed to perturba 3 -dimensional image from a class.
We separate each class’ AR process with a newline.
Note thateach 3 × 3 array has a zero in the final entry because there are only 8 AR coefficients when operating 3 × window.
[Table] [Table]