Attacking Few-Shot Classiﬁers with Adversarial Support Poisoning
Abstract
This paper examines the robustness of deployed few-shot meta-learning systems when they are fed an imperceptibly perturbed few-shot dataset, showing that the resulting predictions on test inputs can become worse than chance.
This is achieved by developing a novel attack, Adversarial Support Poisoning or ASP, which crafts a poisoned set of examples.
When even a small subset of malicious data points is inserted into the support set of a meta-learner, accuracy is signiﬁcantly reduced.
We evaluate the new attack on a variety of few-shot classiﬁcation algorithms and scenarios, and propose a form of adversarial training that signiﬁcantly improves robustness against both poisoning and evasion attacks.
1. Introduction
Standard deep learning approaches suffer from poor sample efﬁciency (Krizhevsky et al., 2012) which is problematic in tasks where data collection is difﬁcult or expensive.
Recently, few-shot learners have been developed which address this shortcoming by supporting rapid adaptation to a new task using only a few labeled examples (Finn et al., 2017;
Snell et al., 2017).
This success has meant that few-shot learners are becoming more attractive for real-life and increasingly sensitive applications where the repercussions of conﬁdently-wrong predictions are severe.
Examples include clinical risk assessment (Sheryl Zhang et al., 2019), glaucoma diagnosis (Kim et al., 2017), identiﬁcation of diseases in skin lesions (Mahajan et al., 2020) and tissue slide annotation in cancer immuno-therapy biomarker research (Lahiani et al., 2018).
As few-shot learners gain popularity, it is essential to understand how robust they are and whether there are potential avenues for their exploitation.
It is well known that standard classiﬁers are vulnerable to inputs that have been purposefully modiﬁed in a minor way to cause incorrect predictions (Biggio & Roli, 2017).
Such examples may be presented to a model either at test time, called evasion attacks (Biggio et al., 2017) or adversarial examples (Szegedy et al., 2014), or at training time, which is referred to as poisoning (Newsome et al., 2006; Rubinstein et al., 2009).
While previous work has considered evasion attacks or query attacks , in the context of few shot learners (Goldblum et al., 2019; Yin et al., 2018), data poisoning attacks have not been studied and are the focus of this paper.
Data poisoning attacks are of particular relevance in the few-shot learning setting for two reasons.
First, since the datasets are small, a handful of poisoned patterns might have a signiﬁcant effect.
Second, many applications of few-shot learning require labeled data from users to adapt the system to a new task, essentially providing a direct interface for outsiders to inﬂuence the model’s behaviour.
Before detailing the key contributions of the paper, we introduce the lexicon of few-shot learning.
During training, few-shot learners are typically presented with many different tasks.
The model must learn to perform well on each task, hopefully arriving at a point where it can adapt effectively to a new task at test time.
At test time, the model is presented with an unseen task containing a few labeled examples, the support set , and a number of unlabeled examples to classify, called the query set .
The paper makes the following contributions:
We deﬁne a novel attack on few-shot classiﬁers, called Adversarial Support Poisoning or ASP, which applies coordinated adversarial perturbations to the support set that are calculated to minimize model accuracy over a set of query points.
To the best of the authors’ knowledge, this is the ﬁrst work considering the impact of poisoning attacks on trained few-shot classiﬁers.
We demonstrate that few-shot classiﬁers are surprisingly vulnerable to our attack.
ASP is more effective than the baselines considered, and generalizes well, i.e. the compromised classiﬁer is highly likely to be inaccurate on a randomly sampled query set from the task domain.
We show the effectiveness of our approach against
ProtoNets (Snell et al., 2017) on the challenging M ETAD ATASET (Triantaﬁllou et al., 2020) benchmarks.
We consider attacks against other models in the appendix.
We propose a form of adversarial training for few-shot learners that signiﬁcantly reduces the effect of both poisoning and evasion attacks.
Section 2 provides the necessary background, Section 3 introduces the ASP attack, Section 4 presents the experimental results and Section 5 concludes the paper.
2. Background
We focus on image classiﬁcation, denoting input images by x ∈ R ch × W × H where W is the image width, H the image height, ch the number of image channels and image labels y ∈ { 1 , . . .
, C } where C is the number of image classes.
We use bold x and y to denote sets of images and labels.
2.1. Meta-Learning
We consider the few-shot image classiﬁcation scenario using a meta-learning approach.
Rather than a single, large dataset
D , we assume access to a dataset D = { τ t } Kt =1 comprising a large number of training tasks τ t , drawn i.i.d.
from a distribution p ( τ ) .
An example task is shown in Fig.
A.1.
The data for a task consists of a support set D S
= { ( x n , y n ) } Nn = comprising N elements, with the inputs x n and labels y n observed, and a query set D Q = { ( x ∗ m , y ∗ m ) }
Mm =1 with
M elements for which we wish to make predictions.
We may use the shorthand D S
= { x , y } and D Q = { x ∗ , y ∗ } .
The meta-learner g takes as input the support set D S and produces task-speciﬁc classiﬁer parameters ψ
= g ( D S ) which are used to adapt the classiﬁer f to the current task.
The classiﬁer can now make task-speciﬁc predictions f ( x ∗ , ψ = g ( D S )) for any test input x ∗ ∈ D Q .
Here the inputs x ∗ are observed and the labels y ∗ are only observed during meta-training (i.e. training of the meta-learning algorithm).
Note that the query set examples are drawn from the same set of labels as the examples in the support set.
The majority of modern meta-learning methods employ episodic training (Vinyals et al., 2016), as detailed in A.2.1.
At metatest time, the classiﬁer f is required to make predictions for query set inputs of unseen tasks.
Often, the meta-test tasks will include classes that have not been seen during meta-training, and D S will contain only a few observations.
The canonical example for modern gradient-based few-shot learning systems is MAML (Finn et al., 2017).
Another widely used class of meta-learners are amortized-inference or black box based approaches e.g,
V ERSA (Gordon et al., 2019) and CNAP S (Requeima et al., 2019a).
In these methods, the task-speciﬁc parameters ψ are generated by one or more hyper-networks , g (Ha et al., 2016).
A special case of this approach is Prototypical Networks (ProtoNets) (Snell et al., 2017) which is based on metric learning and employs a nearest neighbor classiﬁer.
Our paper focuses mainly on
ProtoNets, but we also consider attacks against MAML and
CNAP S in the appendix.
2.2. Threat Model
The threat model may be summarized in terms of the adversary’s goal, knowledge and capabilities.
In this work, we develop poisoning attacks that degrade the model availability (i.e. affect prediction results indiscriminately such that it is useless) (Jagielski et al., 2018).
We assume the adversary has knowledge of the model’s internal workings — including gradients and other internal state information.
We also assume that they can access enough data to be able to form a query set.
These assumptions will be loosened in future work.
We allow the attacker to manipulate some fraction of the support set and further constrain pattern modiﬁcations to be imperceptible (i.e. within some (cid:15) of the original image, measured using the (cid:96) ∞ norm).
3. Adversarial Attacks on Few-Shot Learners
Consider a few-shot learning system that has been trained and deployed as a service.
A malicious party could perpetrate query or poison attacks, as described below:
Query Attack
The attacker may want the adapted classiﬁer to misclassify a speciﬁc input image.
This corresponds to solving arg max δ L ( f ( x ∗ + δ, g ( x , y )) , y ∗ ) .
Refer to
Appendix A.4 for details.
These kinds of attacks are essentially evasion attacks as considered in Biggio et al.
(2017), which we refer to as query attacks in the context of few-shot learners.
Many algorithms can be used to generate adversarial examples (Madry et al., 2017; Carlini & Wagner, 2017;
Chen et al., 2017).
Query attacks have been perpetrated successfully against few-shot learners (Goldblum et al., 2019;
Yin et al., 2018), but are not the main focus of this work.
ASP
The attacker may want the system to fail on any query image, as described in Section 2.2.
The attacker will achieve this by computing a perturbed support set ˜ D S
= { ˜ x , y } whose inputs are optimized to fool the system on a speciﬁc query set, which we call the seed query set , with the goal of generalizing to unseen query sets.
This corresponds to solving arg max δ L ( f ( x ∗ , g ( x + δ , y )) , y ∗ )
such that (cid:107) δ (cid:107) ∞
<
(cid:15) , where D Q = { x ∗ , y ∗ } denotes the seed query set and (cid:15) is the maximum size of the perturbation.
Refer to Algorithm 1 for details.
We call this novel few-shot learner attack Adversarial Support Poisoning , or simply
ASP.
Our attack is a poisoning attack, since the attacker is manipulating data that the model will use to do inference.
However, it is important to note the the attack is perpetrated at meta-test time, after the meta-learner has already been meta-trained.
In contrast to a query attack, which only considers the model’s behaviour on a single point at a time,
ASP attacks allow the adversarial optimization function to incorporate information regarding the model’s behaviour on the entire query set.
In real settings, an attacker might design an attack on their own query set, hoping it will generalize to other, unseen queries.
The ability to generalize may depend on M , the size of the seed query set.
Without loss of generality, we utilize Projected Gradient Descent (PGD) (Madry et al., 2017) with an (cid:96) ∞ norm to generate perturbed support sets because it is effective, simple to implement, and can be extended to handle a set of inputs.
Algorithm
Require:
I min , I max : Min/Max image intensity, γ : Step size
L : Number of iterations, (cid:15) : Perturbation amount
D S ≡ { x , y } , D Q ≡ { x ∗ , y ∗ } , L ≡ cross-entropy 1: procedure PGDS( D S , D Q , f, g ) 2: δ ∼ U ( − (cid:15), (cid:15) ) 3: ˜ x ← clip ( x + δ , I min , I max ) 4: for i ∈ 1 , ..., L do 5: δ ← sgn ( ∇ ˜ x L ( f ( x ∗ , g (˜ x , y )) , y ∗ )) 6: ˜ x ← clip (˜ x + γ δ , I min , I max ) 7: ˜ x ← x + clip (˜ x − x , − (cid:15), (cid:15) ) 8: end for 9: return ˜ x 10: end procedure
4. Experiments
The experiments presented in the main body of the paper are carried out on ProtoNets using the challening M ETAD ATASET benchmark (refer to Appendix A.2 and for details on the meta-data training protocols and M ETA -D ATASET ).
Additional results against other models are presented in the appendix.
In M ETA -D ATASET , task support sets may be large — up to 500 images across all classes.
In a realistic scenario, an attacker would not likely be able to perturb all the images in such a large support set.
We thus perturb only 20 % of the images in the support set, which is generally considered the upper limit of manipulated patterns in conventional poisoning approaches (Jagielski et al., 2018).
For all experiments, we consider the classiﬁer’s performance averaged over 500 randomly generated tasks.
Each task is composed of a support set, a seed query set and up to 50 unseen query sets used for attack evaluation (some
M ETA -D ATASET benchmarks do not have sufﬁciently many patterns available to form 50 query sets).
The unseen query sets are all disjoint from the seed query set to avoid information leakage.
For each task, we generate an adversarial support set using the original support set and corresponding seed query set.
The adversarial support set is then evaluated on the task’s unseen query sets.
We refer to the average classiﬁcation accuracy on the seed query sets as the ASP
Speciﬁc attack accuracy, and when evaluating the attack on unseen query sets we refer to it as the ASP General attack accuracy.
As a baseline, we also consider a swap attack , using the task’s support set and query set to generate a query attack, then “swap” the role of the adversarial query set by using it as a support set.
The swap attack is evaluated on unseen query sets.
Attack Efﬁcacy Fig.
1 shows the relative decrease in accuracy of ProtoNets on M ETA -D ATASET (unnormalized results are in A.6).
We compute the percentage relative decrease in classiﬁcation accuracy as follows: 100% × ( a clean − a attack ) /a
clean where a clean is the clean classiﬁcation accuracy before the attack, and a attack is the classiﬁcation accuracy after the attack.
ASP Speciﬁc is highly effective on all datasets except MNIST, which is the easiest classiﬁcation problem in the large-scale suite because there are only ten classes and the input images are simplistic.
The
ASP attack signiﬁcantly impacts classiﬁcation accuracy, easily out-performing the Swap baseline, in spite of the fact that only 20% of the support set shots are poisoned.
Our results demonstrate that an attacker using ASP could cripple a few-shot learning system in a realistic scenario – with a challenging dataset and a limited number of perturbed patterns – far more effectively than with a simple swap attack.
Results when more than 20% of the support set is poisoned are presented in Fig.
A.4 of the appendix, which shows that the attack predictably becomes even stronger.
The appendix also contains similar results when perpetrating attacks against CNAP S , a more sophisticated meta-learning system, in Table A.7.
Fraction of Poisoned Patterns We consider the importance of the number of perturbed support set points along two dimensions: specifying the fraction of classes that are adversarially perturbed and, within those classes, how many of the shots are poisoned.
These and further experiments are conducted on mini ImageNet Vinyals et al.
due to computational cost.
Fig.
2 ASP shows the relative drop in 5-way classiﬁcation accuracy for ProtoNets as a function of the number of poisoned classes and shots in a support set.
As expected, the attack strength increases as the number of poisoned shots are increased, though perturbing even one point causes a signiﬁcant drop in accuracy.
The attack is stronger when the poisoned points are spread across classes, e.g. perturbing just one image from every class causes 72% accuracy drop, whereas perturbing all ﬁve shots of one class only causes 26% accuracy drop.
For comparison, Fig.
Swap shows the effect of performing a swap attack with
[Figure] [Figure]
varying fractions of poisoned classes and shots.
Poisoning the support set with a single malicious point, using either an adversarial support pattern or an adversarial query pattern achieves similar drops in model accuracy.
However, increasing the number of poisoned shots for a particular class is signiﬁcantly more effective for ASP than the swap attack because the adversarial support set’s joint optimization allows for collusion among its images.
We conﬁrm this with an additional experiment shown in Fig.
2 Subselected , where we generate an adversarial support set with all patterns poisoned by ASP, then select subsets of these perturbed patterns to poison the original support set.
Even though the ASP and Subselected attacks have the same fraction of poisoned patterns, ASP performs signiﬁcantly better as it is jointly optimized for a speciﬁc fraction to be poisoned, whereas the patterns in Subselected are separated from their colluding images.
We conclude that ASP is more effective than attacks that are not set-based as it enables collusion among the poisoned instances.
Adversarial Training To defend against attacks on fewshot classiﬁers, we adversarially train two different models (Goodfellow et al., 2014; Madry et al., 2017) .
The ﬁrst (AS) corrupts all elements of the training support sets with an ASP attack.
The second (AQ) (Goldblum et al., 2019) corrupts all the elements of the training query sets with a query attack.
In order to make the adversarially trained models robust against a variety of attack settings, for each task, the attacks choose (cid:15) ∼ U [0 . 025 , 0 . 05] , L ∼ U [5 , 100] ,
[Figure]
and γ = 3 (cid:15)/L .
Due to the computational cost of training meta-learners, we present results for MAML in 5-way 1shot and 5-way 5-shot scenarios.
Fig.
3 and Fig.
A.8 show that accuracy of models that are not adversarially trained (Natural) decreases signiﬁcantly as (cid:15) is increased and that adversarial training (AS and AQ) mitigates the loss in accuracy to a signiﬁcant extent.
Interestingly, AS outperforms
AQ for all models for both ASP and query attacks, even though the training was intended to mitigate ASP attacks.
Both adversarially trained models are less accurate on clean data, as has been observed in Raghunathan et al.
(2019), resulting in a trade-off between robustness to attacks and model performance.
5. Conclusions
We have introduced Adversarial Support Poisoning, an attack against trained few-shot classiﬁers at meta-test time.
We showed that this attack is very effective, causing predictions to become worse than chance.
We also devised a new form of adversarial training as a defence against ASP.
Future work will consider black box and transfer ASP attacks.
A CKNOWLEDGMENTS
This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service ( www. hpc.cam.ac.uk ) funded by EPSRC Tier-2 capital grant
EP/P020259/1.
A. Appendix
The additional materials presented here are organized as follows: Appendix A.1 discusses related work; Appendix A. provides details regarding meta-learner training protocols and the datasets considered in the paper; Appendix A. summarizes the types of attacks under consideration; Appendix A.4 provides the pseudocode for a query attack against a few-shot learner and shows the algorithm generates results consistent with existing literature; Appendix A. provides further experimental details and tuning results for
ProtoNets; Appendix A.6 provides the unnormalized results for Fig.
1 as well as additional results performing ASP attacks against CNAP S on the M ETA -D ATASET benchmark;
Appendix A.7 provides additional attack results in smallscale scenarios using mini ImageNet against both MAML and Protonets; ﬁnally, Appendix A.8 introduces backdoor attacks and shows that ASP can also be used to perpetrate such attacks with high success rates.
A.1. Related Work
While there has been previous work on adversarial attacks against few-shot learning systems (Goldblum et al., 2019;
Yin et al., 2018), attacks that poison the support set have received little attention.
Goldblum et al.
(2019) devise a technique called adversarial querying (AQ) which signiﬁcantly improves robustness against query attacks.
They do not evaluate robustness against support set attacks at metatest time, though they do test attacking the support set along with the query set during meta-training.
Yin et al.
(2018) also describe a meta-training regime to increase robustness against query attacks, but only consider the relatively weak
Fast Gradient Sign Method attack (Goodfellow et al., 2014).
Edmunds et al.
(2017) explore the transferability of query attacks between tasks on meta-trained models and ﬁnd that the attacks are indeed highly transferable, though their experiments were restricted to the MAML algorithm and the relatively simple Omniglot dataset.
In this work, we focus on the effects of support set poisoning on the performance of few-shot image classiﬁers using a variety of learning algorithms and challenging datasets.
A.2. Few-shot Learner Meta-training Protocols
In the following, the meta-training protocols for the various few-shot learners used in the experiments including MAML,
ProtoNets, and CNAP S will be detailed.
A.2.1. E PISODIC T RAINING
There has been an explosion of meta-learning based fewshot learning algorithms proposed in recent years.
For an in-depth review see Hospedales et al.
The majority of modern meta-learning methods employ episodic training 𝑁
=    𝒙 𝒚 stopwatch clock watch meter 𝑀 =
𝒙 ∗ 𝒚 ∗ meter watch stopwatch clock
S upp o r t S e t
Q u e r y
S e t
(Vinyals et al., 2016).
During meta-training, a task τ is drawn from p ( τ ) and randomly split into a support set D S and query set D Q .
Fig.
A.1 depicts an example few-shot classiﬁcation task.
The meta-learner g takes as input the support set D S and produces task-speciﬁc classiﬁer parameters ψ
= g ( D S ) which are used to adapt the classiﬁer f to the current task.
The classiﬁer can now make task-speciﬁc predictions f ( x ∗ , ψ = g ( D S )) for any test input x ∗ ∈ D Q .
Refer to Clean in Fig.
A.2.
A loss function L ( f ( x ∗ , ψ ) , y ∗ ) then computes the loss between the predictions for the label f
( x ∗ , ψ ) and the true label y ∗ .
Assuming that L , f , and g are differentiable, the meta-learning algorithm can then be trained with stochastic gradient descent by back-propagating the loss and updating the parameters of f and g .
A.2.2. D ATASETS mini
ImageNet mini ImageNet is a subset of the larger Imagenet dataset (Russakovsky et al., 2015) created by Vinyals et al.
It consists of 60,000 color images that is subdivided into 100 classes, each with 600 instances.
The images have dimensions of 84 × 84 pixels.
Ravi & Larochelle (2017) standardized the 64 training, 16 validation, andtest class splits.
mini ImageNet has become a defacto standard dataset for benchmarking few-shot image classiﬁcation methods with the following classiﬁcation task conﬁgurations: (i) 5-way, 1-shot; (ii) 5-way, 5-shot.
M ETA -D ATASET
M ETA -D ATASET (Triantaﬁllou et al., 2020) is composed of ten (eight train, two test) image classiﬁcation datasets.
We augment Meta-Dataset with three additional held-out datasets: MNIST (LeCun et al., 2010),
CIFAR10 (Krizhevsky & Hinton, 2009), and CIFAR (Krizhevsky & Hinton, 2009).
The challenge constructs few-shot learning tasks by drawing from the following distribution.
First, one of the datasets is sampled uniformly; second, the “way” and “shot” are sampled randomly according to a ﬁxed procedure; third, the classes and support / query instances are sampled.
Where a hierarchical structure exists in the data (ImageNet or Omniglot), task-sampling respects the hierarchy.
In the meta-test phase, the identity of the original dataset is not revealed and the tasks must be treated independently (i.e. no information can be transferred between them).
Notably, the meta-training set comprises a disjoint and dissimilar set of classes from those used for meta-test.
M ETA -D ATASET is presently, the ”gold standard” for evaluating few-shot classiﬁcation methods.
Full details are available in Triantaﬁllou et al.
In our experiments, we excluded the Omniglot, Textures,
Fungi, and Trafﬁc Signs datasets from evaluation because their test splits are too small to allow for a fair assessment of the attack’s generalization, even though the attacks reduced the classiﬁcation accuracy on those datasets to approximately zero in the ASP Speciﬁc case.
A.2.3. MAML M ETA -T RAINING P ROTOCOL
We meta-trained our implementation of MAML on mini ImageNet with identical network conﬁguration, hyperparameters, and training protocol as prescribed in Finn et al.
The meta-trained models attained the following accuracy:
mini ImageNet 5-way, 1-shot: 47 .
2 ± 1 .
7% mini ImageNet 5-way, 5-shot: 61 . 3 ± 0 .
A.2.4. P ROTO N ETS M ETA -T RAINING P ROTOCOL
We meta-trained our implementation of ProtoNets on mini ImageNet with identical network conﬁguration, hyperparameters, and training protocol as prescribed in Snell et al.
The meta-trained models attained the following accuracy:
mini ImageNet 5-way, 1-shot: 46 . 8 ± 0 .
6% mini ImageNet 5-way, 5-shot: 65 . 1 ± 0 .
For M ETA -D ATASET , we meta-trained ProtoNets using the code from Requeima et al. (2019b) with FiLM feature adaptation.
We made modiﬁcations to the code to enable various adversarial attacks.
The meta-trained model attained the following results:
ilsvrc 2012: 55 .
1 ±
1 , omniglot: 90 . 8 ± 0 .
6 , aircraft: 82 . 3 ± 0 .
6 , cu birds: 74 . 0 ± 0 .
9 , dtd: 63 .
4 ± 0 .
7 , quickdraw: 75 . 3 ± 0 . 8 , fungi: 44 .
6 ± 1 . 0 , vgg ﬂower: 90 . 3 ± 0 .
5 , trafﬁc sign: 67 . 9 ± 0 .
8 , mscoco: 40 .
8 ± 1 . 0 , mnist: 91 .
4 ± 0 .
5 , cifar10: 72 .
5 ± 0 .
8 , cifar100: 58 .
4 ± 1 . 0 .
A.2.5. CNAP S M ETA -T RAINING P ROTOCOL
For all the CNAP S experiments, we use the code provided by the the CNAP S authors (Requeima et al., 2019b).
We made modiﬁcations to the code to enable various adversarial attacks and used FiLM feature adaptation.
We follow an identical dataset conﬁguration and meta-training process as prescribed in Requeima et al.
(2019b).
A.3. Attack Summary
We summarize the types of attacks we perform in Fig.
A.2.
The ﬁrst scenario, Clean , illustrates how the meta-learner g performs a test-time task, taking the support set D S as input to produce parameters ψ
= g ( D S ) which are used to adapt the classiﬁer f to the task.
The classiﬁer makes task-speciﬁc predictions f ( x ∗ , ψ = g ( D S )) for any test input x ∗ ∈ D Q .
ASP and Query illustrate the ASP and
Query attacks as discussed in Section 3 and Appendix A.4.
Swap illustrates a swap attack, which is used as a baseline comparison for ASP, where a set of images are perturbed with a query attack and then inserted into the support set.
Query attacks are typically cheaper to compute, since they do not require back-propagation through the meta-learner, so it is an important baseline to consider.
Label Shift is a simple attack on the support set which involves mislabelling the support set images by shifting the true label index by one in a modulo arithmetic fashion.
We consider systematic mislabeling in this way to be a strong attack for comparison, though this “attack” would be easily detected by inspection.
We thus only consider the label shift baseline for the small scale results presented in Appendix A.7.
A.4. Query Attacks
We present our algorithm for performing query attacks with
PGD in Algorithm A.1.
Using this algorithm, we attack
MAML and ProtoNets with settings that match the experiments by (Goldblum et al., 2019) to ensure that our attack performs approximately the same, as expected.
The attack settings used are L = 20 , (cid:15)
= 8255 , γ = 2255 , with an untargeted loss function.
Our results are shown in Table A.1 and the relevant results from (Goldblum et al., 2019) are shown in Table A.2.
Our models perform similarly when presented with clean data and when attacked using PGD, as expected
[Table]
A.5. Further Experimental Details
In our experiments all the input images were re-scaled to have pixel values between − 1 and 1 .
We considered perturbations using the (cid:96) ∞ norm, on a scale of [ − 1 , 1] , so that (cid:15) = 0 .
1 corresponds to allowing ± 10% or an absolute change of ± 0 .
2 to the intensity of each pixel in an image.
𝑔 𝑓 𝐷 𝑆 = {𝒙,𝒚} ෥𝒙 ∗ 𝒚 ∗ 𝝍
Query ෥𝒙 ∗ = PGDQ(𝐷 𝑆 ,𝐷 𝑄, 𝑓,𝑔) 𝑔 𝑓 ෩𝐷 𝑆 = {෥𝒙,𝒚} 𝒙 ∗ 𝒚 ∗ 𝝍
ASP ෥𝒙 = PGD𝑆(𝐷 𝑆 ,𝐷 𝑄, 𝑓,𝑔) 𝑔 𝑓 𝐷 𝑆 = {𝒙,𝒚} 𝒙 ∗ 𝒚 ∗ 𝝍
Clean 𝑔 𝑓 ෩𝐷 𝑆 = {෥𝒙 ∗ ,𝒚 ∗ } 𝒙 𝒚 𝝍
Swap ෥𝒙 ∗ =
PGDQ(𝐷 𝑆 ,𝐷 𝑄, 𝑓,𝑔) 𝑔
𝑓 ෩𝐷 𝑆 = {𝒙, ෥𝒚} 𝒙 ∗ 𝒚 ∗ 𝝍
Label Shift ෤𝑦 𝑛 = 𝑦 𝑛+1 mod 𝐶
Require: size
D S ≡ { x , y }
D Q ≡ { x ∗ , y ∗ } (cid:46)
We use cross-entropy loss for L .
procedure PGDQ( D S , D Q , f, g ) 2: δ ∼ U ( − (cid:15), (cid:15) ) 3: ˜ x ∗ ← clip ( x ∗ + δ , I min , I max ) 4: for n ∈ 1 , ..., L do 5: δ ← sgn ( ∇ ˜ x ∗ L ( f (˜ x ∗ , g ( x, y )) , y ∗ ) 6: ˜ x ∗ ← clip (˜ x ∗ + γ δ , I min , I max ) 7: ˜ x ∗ ← x + clip
(˜ x ∗ − x ∗ , − (cid:15), (cid:15) ) 8: end for 9: return ˜ x ∗ 10: end procedure
[Table]
We calculated the perturbation step size γ to depend on (cid:15) and the maximum number of iterations, so that γ = r (cid:15)L , where r is a scaling coefﬁcient.
We observed that the optimal values for r depend on the numbers of shots, and varies with (cid:15) and
L .
Our experiments here performed both targeted and untargeted attacks.
When perpetrating ASP attacks, we consider two variations of the loss function: all , in which the entire query set is used; and single , in which a single, random point in the query set is chosen for the loss calculation at each attack iteration, with the intention that the additional stochasticity may prevent the attack from getting stuck in local optima.
We found that the all strategy performed best when combined with a targeted attack, whereas the single strategy worked better in combination with an untargeted attack.
Unless otherwise speciﬁed, we use the targeted, all loss strategy.
The ASP results presented in the main body of the paper are all targeted attacks using the all loss strategy.
Example images from an ASP attack are shown in Fig.
A.3.
Protonets Tuning The results of our tuning experiments are provided for ProtoNets on mini ImageNet in Table A.
- Table A.5.
In general, larger values of r (i.e. larger step sizes) performed better as the number of PGD iterations increased.
Although the single loss strategy did not perform well for 1-shot classiﬁcation, its performance increased at higher shots, often out-performing the all strategy for sufﬁciently large numbers of PGD iterations, even though the
Speciﬁc accuracy did not go to 0 .
The all strategy performed signiﬁcantly better than single when L was low.
A.6. Additional Large-Scale Attack Results
Here, we present the unnormalized numbers for Fig.
1 in
Table A.6.
We also present results for a similar attack, perpetrated against CNAP S in Table A.7, showing that our attack is also effective against CNAP S in a large-scale scenario.
Fig.
A.4 shows the effectiveness of both ASP and swap attacks increase as the fraction of poisoned shots within a
[Table] [Table]
[Table] [Table]
class increases.
It further demonstrates the superiority of
ASP compared to the swap attack in the large-scale few-shot learning setting.
A.7. Additional Small-Scale Experiments
Small-Scale Results Fig.
A.5 depicts the relative decrease in 5-way classiﬁcation accuracy due to a variety of attacks for two values of (cid:15) on mini ImageNet.
As in the main body of the paper, we compute the percentage relative decrease in classiﬁcation accuracy as follows: 100% × ( a clean − a attack )
/a
clean where a clean is the clean classiﬁcation accuracy before the attack, and a attack is the classiﬁcation accuracy after the attack.
We include two dif-
[Table]
[Figure]
R e l a t i v e D e
c
r
e
a
s
e
i n A cc u r a c
[Figure]
ferent results for the ASP General scenario: ASP General (1x) , for which the seed query set size is the same size as the support set (i.e. M = N ) and ASP General (10x) , where
M = 10 N .
The ASP Speciﬁc attack is 100% effective in all cases, indicating that the adversarial optimization is acting as desired.
The attack’s generalization to unseen query sets, as measured by the ASP General (x10) attack, successfully beats all the baselines.
The ASP General (x1) attack, is noticeably weaker but still beats the Swap baseline on Protonets and the Label Shift baseline on MAML.
Since we are measuring generalization in these scenarios, it is expected that an attack with less data to learn from does not generalize as well.
We note that the results presented in the main body of the paper, on more difﬁcult and realistic problems, do not require large seed query sets to succeed.
The Label
Shift attack causes a large drop in accuracy, but is easily detected by inspection, and so does not provide like-for-like comparison.
Even so, ASP General (10x) is more effective than Label Shift in all scenarios.
From Fig.
A.5, there does not appear to be a difference between ProtoNets and
MAML in terms of robustness to the ASP attack, though
MAML is signiﬁcantly more vulnerable to the swap attack.
For 5-shot classiﬁcation, increasing the perturbation size does not consistently increase the attack’s impact.
This is because the step size γ was tuned for the 1-shot case, but was not optimal for the 5-shot problem.
For consistency, we used the same PGD settings for both in Fig.
A.5.
Since we are primarily interested in degrading model accuracy for unseen inputs, we consider only the ASP General case for the remainder of the results, unless otherwise speciﬁed.
To supplement Fig.
A.5, we provide the unnormalized in
Tables A.8 and A.9, for the 1-shot and 5-shot scenarios, respectively.
All adversarial query points used in the swap attacks achieved approximately 100% fooling rates when presented to the learner as query attacks.
[Table]
[Table]
Fraction of Poisoned Patterns In addition to Fig.
2 in
Section 4 of the paper, we provide similar plots for MAML,
ProtoNets and CNAP S on 5-way, 1-shot and 5-way, 5-shot problems in Figures A.6 and A.7.
Note that for these results,
MAML and ProtoNets were performing classiﬁcation on mini ImageNet, whereas CNAP S was performing classiﬁcation on ILSVRC-2012, which is a more difﬁcult problem.
Adversarial Training for Protonets Fig.
A.8 is the same as Fig.
3 except that the ProtoNets algorithm is used instead of MAML.
The overall trend is consistent between MAML and ProtoNets, though adversarial training is more robust in the MAML case as the AS and AQ accuracy decays to a lesser extent as (cid:15) is increased.
In addition, the difference between AS and AQ accuracy is smaller in the ProtoNets case .
A.8. Backdoor Poisoning Attacks
We also consider targeted backdoor poisoning attacks, where the goal is for a speciﬁc input x ∗ in the query set
[Figure]
l a t i v e D e
c r e a s e i n
A cc
[Figure]
r
[Figure]
c y
[Figure] [Figure]
to be misclassiﬁed as class t .
We assume the attacker can only poison one element, ˜ x , in the support set.
The support set point x that is used to generate ˜ x is chosen from the target class (i e. y = t ).
An attacker has a speciﬁc x ∗ and t in mind, but to evaluate the attack’s feasibility, we randomly choose x , x ∗ and t such that y = t , and so that x ∗ is initially correctly classiﬁed (this removes the case where the classiﬁer assigns the incorrect label to x ∗ , which may inﬂate our results).
The poisoned instance is generated as usual, except that the loss is only calculated with respect to x ∗ , not with respect to the entire seed seed query set, as for
[Figure]
MAML, 5-shot (Clean 61.4%)  5 Classes 21.
CNAPs, 5-shot (Clean 66.7%) the usual ASP attack.
We evaluate the backdoor attack in the Speciﬁc scenario by presenting the model with both the support set and query set used in attack generation (where the support set contains ˜ x instead of x ).
We also evaluate the attack’s ability to generalize (i.e. the General scenario) by inserting ˜ x into a different support set that is randomly generated for the same task.
We call the backdoor attack a success in either the Speciﬁc or General scenario if the targeted instance, x ∗ is classiﬁed as belonging to class t at meta-test time.
Table A.10 shows that ASP backdoor attacks are very successful against the ProtoNets Natural model.
In the 1-shot case, ASP is able to cause the targeted point to be misclassiﬁed as class t more than 99 % of the
[Figure] [Figure]
[Table]
[Figure]
[Table]
time, and more than 85 % in the 5-shot setting.
We conclude that that an ASP backdoor attack is highly likely to succeed, even in the general case, using just one poisoned example inserted into an unknown support set.
For backdoor attacks to remain undetected, the accuracy of the model should be preserved (except for the special, targeted point).
Table A. demonstrates that the overall accuracy of the model did not decrease signiﬁcantly with the introduction of the poisoned input in the 5-shot setting.
For 1-shot, the accuracy drops quite signiﬁcantly because the now incorrect, targeted point forms 20% of the query set, but the accuracy never drops by more than 20% .
The ASP backdoor attack is thus difﬁcult to detect by examining model accuracy.