Long-term Cross Adversarial Training: A Robust Meta-learning Method for Few-shot Classiﬁcation Tasks
Abstract
Meta-learning model can quickly adapt to new tasks using few-shot labeled data.
However, despite achieving good generalization on few-shot classiﬁcation tasks, it is still challenging to improve the adversarial robustness of the metalearning model in few-shot learning.
Although adversarial training (AT) methods such as Adversarial Query (AQ) can improve the adversarially robust performance of meta-learning models, AT is still computationally expensive training.
On the other hand, meta-learning models trained with AT will drop signiﬁcant accuracy on the original clean images.
This paper proposed a meta-learning method on the adversarially robust neural network called Long-term Cross Adversarial Training (LCAT).
LCAT will update metalearning model parameters cross along the natural and adversarial sample distribution direction with long-term to improve both adversarial and clean few-shot classiﬁcation accuracy.
Due to crossadversarial training, LCAT only needs half of the adversarial training epoch than AQ, resulting in a low adversarial training computation.
Experiment results 2 show that LCAT achieves superior performance both on the clean and adversarial few-shot classiﬁcation accuracy than SOTA adversarial training methods for meta-learning models.
1. Introduction
Meta-learning only needs few-shot data that can quickly adapt to even unseen tasks and has been widely applied to many ﬁelds (Haoxiang Wang, 2021). However, recent
Accepted by the ICML 2021 workshop on A Blessing in Disguise: The Prospects and Perils of Adversarial Machine Learning.
Copyright author(s).
The code to reproduce all the experimental results is avail- able research shows that meta-learning based-models are also vulnerable to adversarial examples (Goldblum et al., 2020;
Croce & Hein, 2019).
Adversarial training trained on the large-scale dataset can improve the adversarial robustness of deep neural networks (Rice et al., 2020).
However, how to train on few-shot tasks to robustly defend adversarial examples, fast adapt to other tasks is still challenging.
Recent robust meta-learning work called Adversarial Query (AQ) (Goldblum et al., 2020) demonstrates that adversarial training only during the query step can compromise the superior adversarially robust performance compared to traditional adversarial training (Madry et al., 2017b).
However,
AQ is still computationally expensive training throughout the whole query step.
In addition, AQ will drop signiﬁcant accuracy on the original clean image distribution, which means meta-learning models cannot adapt well to other unseen tasks.
This paper proposes a new robust meta-learning method called Long-term Cross Adversarial Training (LCAT).
In detail, LCAT will make the meta-learning model’s parameter θ cross along the natural and adversarial sample distribution direction.
LCAT is a model-agonist meta-learning method, can adversarially adapt to few-shot classiﬁcation tasks, i.e., perform well on the original clean sample distribution and robust to adversarial examples.
Our solved problem can be formulated as Eq.
(1), where T ( x, y ) is training data distribution.
θ is meta-learning model
f θ ’s parameters.
A is the ﬁne-tuning algorithm during the inner loop for meta-learning.
(cid:15) is the maximum bound for adversarial perturbation.
L is the loss function.
[Equation]
LCAT only needs half of the adversarial training epoch via cross adversarial training and signiﬁcantly improves the robust performance compared to the the-state-of-art adversarial training AQ for meta-learning models.
The proposed method LCAT is illustrated in Fig.
Our contribution can be summarized as follows:
• We propose an adversarially robust meta-learning
✓ ✓ ⇤ r ✓ L i ( f ✓ ,⌧ i ( clean ))
r ✓ L i ( f ✓ ,⌧ i ( adv )) (a) ours ✓ ✓ ⇤ r ✓ L i ( f ✓ ,⌧ i ( adv ))
{+(b) AQ ✓ ✓ 0 ⌧ j ( clean ) ✓ 
+}
⌧
i ( adv ) ✓ ⇤ r ✓ L i ( f ✓ ,⌧ j ( clean ))
r ✓ L i ( f ✓ ,⌧ i ( adv )) r ✓ L i ( f ✓ 0 ⌧j ( clean ) ,⌧ i ( adv )) r ✓ L i ( f ✓ 0 ⌧i ( adv ) ,⌧ j ( clean )) [Figure]
[Figure]
method called LCAT.
LCAT will update meta-learning model parameters cross along the natural and adversarial sample distribution direction with long-term to improve both adversarial and clean few-shot classiﬁcation accuracy.
• LCAT is a model-agonist meta-learning method, which can improve the adversarial robustness of metalearning models.
In addition, LCAT only needs half of the adversarial training epoch compared to AQ via cross adversarial training, resulting in a low adversarial training computation.
• Experimental results show that LCAT improves both clean and adversarial accuracy compared to SOTA method AQ.
i.e. improve 9.7% clean and 2.88% adversarial few-shot classiﬁcation accuracy compared to previous best results in MetaOptNet (Lee et al., 2019) corresponding to 5-way 5-shot on MiniImageNet dataset.
2. Related Work
There’re several methods to train a robust Deep Learning model in normal scenarios rather than a meta-learning model, including defensive distillation (Papernot et al., 2016), adversarial training (Madry et al., 2017a; Su et al., 2016), feature denoising (Xie et al., 2019), TRADES theoretically principled the trade-off between accuracy and robustness (Zhang et al., 2019), and adversarial example detection (Hendrycks & Gimpel, 2016; Xu et al., 2017;
Gong et al., 2017; Grosse et al., 2017).
There are many adversarial robust deep neural network applications, including (Pang et al., 2020; Dong et al., 2020;
Feng et al., 2021; Hendrycks et al., 2020;
Dai et al., 2018; Pang et al., 2021; Dong et al., 2021).
Some works published in recent years discussed the robustness of the meta-learning model from different aspects.
Yin et al.
(2018) introduce a robust variant of MAML (Finn et al., 2017) called ADML, which is the ﬁrst attempt to achieve robustness on the meta-learning model.
However,
ADML needs lots of computational power while the attacker used in their experiments is relatively weak.
Goldblum et al.
(2020) present a method to integrate MAML with adversarial training, called adversarial querying(AQ), which is the state-of-the-art approach in this domain.
We choose AQ as one of the baselines in our experiments.
In (Wang et al., 2021), they build a principled robustness-regularized metalearning framework, which can be treated as
a generalized
AQ model.
3. Proposed Method: LCAT
Our goal is to make meta-learning models perform well both on clean and adversarial few-shot classiﬁcation tasks.
We now formulated the problem we want to solve, which can be seen in Eq.
Cross adversarial training The proposed method called
Long-term cross adversarial training (LCAT) is illustrated in Fig.
The LCAT across along the direction of the clean and adversarial direction.
In the clean meta-update step, the LCAT will optimize the meta-learning model f θ in the clean batch of tasks.
This step aims to give the meta-model a good initial start which quickly adapts to clean few-shot classiﬁcation tasks.
In the adversarial meta-update step, the
LCAT will optimize f θ in the adversarial batch of tasks, which can help the meta-learning model against adversarial attacks.
It is important to notice that LCAT is model-agnostic.
Our adversarial training method can help the meta-learning model improve signiﬁcance accuracy on the clean image distribution and help the meta-learning model improve adversarially robust accuracy with a low computation compared to a previous SOTA adversarial training AQ.
The method of LCAT training method is as follows.
STEP 1 Generate a batch of tasks according to the distribution of tasks ρ ( τ ) (Goldblum et al., 2020).
[Figure]
STEP 2 Fine-tune each task { τ i } 0 ≤ i<n based on the ﬁne- tuning algorithm
A ( θ, T ) .
STEP 3 Make model parameter theta update along with the clean sample space with long-term T based on Eq.
[Equation]
Where θ is the meta-leaning model parameter, and f ( j ) θ is the meta-learning model at j th term.
λ is the learning rate.
STEP 4 Compute adversarial examples of tasks { τ (cid:48) i } based on attacker (Madry et al., 2017a) within a (cid:15) ball and then make the model update in the space of adversarial examples for long-term T based on Eq.
[Equation]
The detail of the LCAT training method is shown in Alg. 1.
Why need long-term cross training? Short-term cross ad-
versarial training cannot give suitable initial parameters for meta-learning models, as shown in Fig.
Long-term cross adversarial training can make the model’s parameters go deeper in the adversarial sample space, resulting in good adversarial training.
The model can not be well trained in the adversarial sample space corresponding to short-term adversarial training.
In Tab. 2, we test the short-term adversarial training.
In detail, we set ten epochs as a loop, the ﬁrst nine epochs are natural training, and the last epoch is adversarial training.
Fig.
2 and Tab.
2 show the training results.
The test results show that the model trained with short-term has low adversarial accuracy compared to longterm, cannot sufﬁciently defend adversarial samples.
In addition, to avoid long-term cross-training appearing in the clean and adversarial sample space to go deeper and lead to poor performance of the model in few-shot classiﬁcation, we also added the denoise module (Xie et al., 2019) for the meta-learning model.
The Sec. 5 shows the effectiveness of the denoise module. Algorithm
Input:
Meta-learning model, f θ , learning rate, λ , ﬁne-tune algorithm A, tasks distribution ρ ( τ ) , long-term T ; 1: Initialize all parameters; 2: while not done do 3:
Sample batch of tasks, { τ i } 0 ≤ i<n ; 4: for j = 0 ; j < T ; j + + do 5: for i = 0 ; i < n ; i + + do 6: θ
=
A ( θ, τ i ) , ﬁne tine on task τ
i ; 7:
Compute updated gradient ∇ θ L ( f ( j ) θ , τ i ) ; 8: end for 9: end for 10: θ ← θ − λn (cid:80) 0 ≤ j<T (cid:80) τ
i ∼ ρ ( τ ) ∇ θ L ( f ( j ) θ , τ i ) ; 11: for j (cid:48) =
T ; j (cid:48) <
2 T ; j (cid:48)
+ + do 12: for i = 0 ; i < n ; i + + do 13: θ =
A ( θ, τ i ) , ﬁne tine on task τ
i ; 14:
Generate adversarial batch of tasks, { τ (cid:48) i } 0 ≤ i<n ∇ θ L ( f θ , τ (cid:48) i ) ; 15:
Compute updated gradient ∇ θ L ( f ( j (cid:48) )
θ , τ (cid:48) i ) ; 16: end for 17: end for 18: θ ← θ − λn (cid:80) T ≤ j (cid:48) <
2 T (cid:80) τ (cid:48)
i ∼ ρ ( τ (cid:48) )
∇ θ L ( f ( j (cid:48) )
θ , τ (cid:48) i ) ; 19: end while
4. Experiments 4.1. Setup
Baselines To test the effectiveness of LCAT, we compared our adversarial training with 1)
Adversarial Training (AT) (Madry et al., 2017b), a directly robust adversarial training method applied in the meta-learning method.
The Adversarial Query (AQ) (Goldblum et al., 2020) is a
SOTA adversarial training method that only needs adversarial training in the query step.
3) Adversarial Query plus
TRADES (Zhang et al., 2019) loss function can be a trade between natural accuracy and robust accuracy.
We call this method AQ+TRADES which is suitable as a SOTA method compared to LCAT.
The learning rate is set to 0 .
The epoch and the batch size are set to 50 and 8 .
We use Adam (Kingma & Ba, 2014) to optimize the meta-learning models, which has been proved to obtain better performance than SGD (see Appendix A.3). [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We follow the experimental setting of AQ in (Goldblum et al., 2020), training the state-of-the-art metalearning models including PROTONET (Snell et al., 2017a),
R2D2 (Bertinetto et al., 2018a) , and MetaOptNet ( ResNet12 as backbone (He et al., 2016)).
We conduct all the experiments on the Pytorch with Ubuntu 20.04 and GPU
RTX3090.
4.2. Experimental results
The experimental results can be seen in Tab. 1.
Due to the space limit, training details and other results can be seen in Appendix A.1.
To sum up, our LCAT achieves superior performance both on the clean and adversarial few-shot classiﬁcation accuracy than SOTA adversarial training methods for meta-learning models.
For example, we improve 9.7% clean few-shot classiﬁcation accuracy compared to previous best results in MetaOptNet and improve 2.88% adversarial few-shot classiﬁcation accuracy compared to previous best results in MetaOptNet corresponding to 5-way 5-shot on
MiniImageNet dataset.
5. Ablation Study
We also conducted the following experiments: 1) short-term cross adversarial training (SCAT): we set ten epochs as a loop, the ﬁrst nine epochs are norm training, and the last epoch is adversarial training (T=1).
2) Long-term cross adversarial training (LCAT): we set ten epochs as a loop, the ﬁrst ﬁve epochs are norm training, and the last ﬁve epochs are adversarial training (T=5).
3) LCAT with/without denoise (w/o denoise).
The results are shown in Tab. 2.
which can be seen that our LCAT compare to short-term and (w/o denoise ) achieve supervisor performance.
In addition, we also test the adversarial robustness of metalearning models trained with LCAT.
With the increase of attack step, PROTONET trained with LCAT still has higher adversarial few-shot classiﬁcation accuracy compared to other adversarial training methods in Fig.
3 (See Appendix
A.4 for R2D2 and MetaOptNet ).
Further investigations about the inﬂuence of TRADES are shown in Tab. 10 in
Appendix A.2.
[Figure]
M ETHODS
[Table]
IME
[Table]
6. Conclusion
In this paper, we propose an adversarially robust metalearning method called LCAT.
LCAT is a model-agonist meta-learning method, which can improve the adversarial robustness of meta-learning models.
LCAT will update meta-learning model parameters cross along the natural and adversarial sample distribution direction with long-term to improve both adversarial and clean few-shot classiﬁcation accuracy.
In addition, LCAT only needs half of the adversarial training epoch compared to AQ via cross adversarial training, resulting in a low adversarial training computation.
In the future, we will establish a more in-depth understanding of the theoretical analysis of LCAT.
A. Appendix
The code to reproduce all the experimental results is avail- able in https://github.com/Gnomeek/Long-term-
Cross-Adversarial-Training . A.1. Experiments result with models trained in 50/ epoch on MiniImageNet, TieredImageNet, CIFAR-FS, and FC
Training details.
We adopt models including PROTONET (Snell et al., 2017b), R2D2 (Bertinetto et al., 2018b), and MetaOptNet (Lee et al., 2019) and dataset including MiniImageNet (Vinyals et al., 2016), TieredImageNet (Ren et al., 2018), CIFARFS (Bertinetto et al., 2018b), and FC100 (Oreshkin et al., 2018).
We follow the experimental setting (Goldblum et al., 2020) that the maximum adversarial perturbation magnitude (cid:15) is 8/255.
In training, the attacker is 7-step PGD with a 2/255 step size.
We use
Adam (Kingma & Ba, 2014) to optimize the meta-learning models with learning 0.1 and 50 epochs.
In testing, we use a 20-step PGD attacker with a 2/255 step size.
D ATASET
[Table]
ETHODS
[Table]
[Table]
[Table]
ATASET
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure]
D ATASET M ETHODS
[Table]
A cc adv
T IME
[Table] cc
[Table]
A cc adv
T IME
[Table] adv
T IME
CIFAR-FS
[Table]
[Table]
[Table]
D ATASETS M ETHODS
[Table]
A.2. Experiments results on applying TRADES at different times
[Table]
A.3. Experiments results on LCAT applied with different optimizer
[Table]
[Figure]
A.4. Extended experiments results on R2D2 and MetaOptNet versus PGD attack at different attack steps
[Figure]
[Figure]