T EST - TIME A DAPTATION FOR B ETTER A DVERSARIAL R OBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
Standard adversarial training and its variants have been widely adopted in practice to achieve robustness against adversarial attacks.
However, we show in this work that such an approach does not necessarily achieve near optimal generalization performance on test samples.
Speciﬁcally, it is shown that under suitable assumptions, Bayesian optimal robust estimator requires test-time adaptation, and such adaptation can lead to signiﬁcant performance boost over standard adversarial training.
Motivated by this observation, we propose a practically easy-toimplement method to improve the generalization performance of adversariallytrained networks via an additional self-supervised test-time adaptation step.
We further employ a meta adversarial training method to ﬁnd a good starting point for test-time adaptation, which incorporates the test-time adaptation procedure into the training phase and it strengthens the correlation between the pre-text tasks in self-supervised learning and the original classiﬁcation task.
Extensive empirical experiments on CIFAR10, STL10 and Tiny ImageNet using several different selfsupervised tasks show that our method consistently improves the robust accuracy of standard adversarial training under different white-box and black-box attack strategies.
I NTRODUCTION
Adversarial Training (AT) (Madry et al., 2018) and its variants (Wang et al., 2019; Zhang et al., 2019) are currently recognized as the most effective defense mechanism against adversarial attacks.
However, AT generalizes poorly; the robust accuracy gap between the training and test set in AT is much larger than the training-test gap in standard training of deep networks (Neyshabur et al., 2017;
Zhang et al., 2017).
Unfortunately, classical techniques to overcome overﬁtting in standard training, including regularization and data augmentation, only have little effect in AT (Rice et al., 2020).
Theoretically, as will be shown in Section 3, the loss objective of AT does not achieve optimal robustness.
Instead, under suitable assumptions, the Bayesian optimal robust estimator, which represents the statistical optimal model that can be obtained from training data, requires test-time adaptation.
Compared with the ﬁxed restricted Bayesian robust estimators, the test-time adapted estimators largely improve the robustness.
Therefore, we should perform the test-time adaptation for each test input to boost the robustness.
To this end, we propose to ﬁne-tune the model parameters for each test mini-batch.
Since the labels of the test images are not available, we exploit self-supervision, which is widely used in the standard training of networks (Chen et al., 2020b;
Gidaris et al., 2018;
He et al., 2020).
Fine-tuning the self-supervised tasks has a high gradient correlation with ﬁne-tuning the classiﬁcation task so that it forms a substitute of ﬁne-tuning the classiﬁcation loss at the inference time.
Thus, we expect minimizing this self-supervised loss function yields better generalization on the test set.
To make our test-time adaptation strategy effective, we need to search for a good starting point that achieves good robust accuracy after ﬁne-tuning.
As will be shown in our experiments, AT itself does not provide the optimal starting point.
We therefore formulate the search for such start point as a bilevel optimization problem.
Speciﬁcally, we introduce a Meta Adversarial Training (MAT) strategy dedicated to our self-supervised ﬁne-tuning inspired by the model-agnostic meta-learning (MAML) framework (Finn et al., 2017).
To this end, we treat the classiﬁcation of each batch of adversarial images as one task and minimize the corresponding classiﬁcation error of the ﬁne-tuned network.
MAT strengthens the correlation between the self-supervised and classiﬁcation tasks so that self-supervised test-time adaptation can further improve robust accuracy.
In order to reliably evaluate our method, we follow the suggestions of (Tramer et al., 2020) and design an adaptive attack that is fully aware of the test-time adaptation.
Using rotation and vertical ﬂip as the self-supervised tasks, we empirically demonstrate the effectiveness of our method on the commonly used CIFAR10 (Krizhevsky et al., 2009), STL10 (Coates et al., 2011) and Tiny
ImageNet (Le & Yang, 2015) datasets under both standard (Andriushchenko et al., 2020; Croce &
Hein, 2020a; Madry et al., 2018) and adaptive attacks in both white-box and black-box attacks.
The experiments evidence that our method consistently improves the robust accuracy under all attacks.
Our contributions can be summarized as follows:
We show that the estimators should be test-time adapted in order to achieve the Bayesian optimal adversarial robustness, even for simple models like linear models.
And the test-time adaptation largely improves the robustness compared with optimal restricted estimators.
We introduce the framework of self-supervised test-time ﬁne-tuning for adversarially-trained networks, showing that it improves the robust accuracy of the test data.
We propose a meta adversarial training strategy based on the MAML framework to ﬁnd a good starting point and strengthen the correlation between the self-supervised and classiﬁcation tasks.
The experiments show that our approach is valid on diverse attack strategies, including an adaptive attack that is fully aware of our test-time adaptation, in both white-box and black-box attacks.
R ELATED W ORK
Adversarial Training.
In recent years, many approaches have been proposed to defend networks against adversarial attacks (Guo et al., 2018;
Liao et al., 2018; Song et al., 2018).
Among them,
Adversarial Training (AT) (Madry et al., 2018) stands out as one of the most robust and popular methods, even under various strong attacks (Athalye et al., 2018; Croce & Hein, 2020a).
AT optimizes the loss of adversarial examples to ﬁnd parameters that are robust to adversarial attacks.
Several variants of AT (Wang et al., 2019; Zhang et al., 2019) also achieved and similar performance to AT
(Rice et al., 2020).
One important problem that limits the robust accuracy of AT is overﬁtting.
Compared with training on clean images, the gap of robust accuracy between the training and test set is much larger in AT (Rice et al., 2020).
Moreover, traditional techniques to prevent overﬁtting, such as regularization and data augmentation, have little effect.
Recently, some methods have attempted to ﬂatten the weight loss landscape to improve the generalization of AT.
In particular, Adversarial Weight Perturbation (AWP) (Wu et al., 2020) achieves this by designing a double-perturbation mechanism that adversarially perturbs both inputs and weights.
In addition, learning-based smoothing can ﬂatten the landscape and improve the performance (Chen et al., 2021b).
Self-supervised Learning.
In the context of non-adversarial training, many self-supervised strategies have been proposed, such as rotation prediction (Gidaris et al., 2018), region/component ﬁlling (Criminisi et al., 2004), patch-base spatial composition prediction (Trinh et al., 2019) and contrastive learning (Chen et al., 2020b; He et al., 2020).
While self-supervision has also been employed in AT (Chen et al., 2020a; Kim et al., 2020; Yang & Vondrick, 2020; Hendrycks et al., 2019), their methods only use self-supervised learning at training time to regularize the parameters and improve the robust accuracy.
By contrast, we propose to perform self-supervised ﬁne-tuning at test time, which we demonstrate to signiﬁcantly improve the robust accuracy on test images.
As will be shown in the experiments, the self-supervised test-time adaptation has larger and complementary improvements over the training time self-supervision.
Test-time Adaption.
Test-time adaptation has been used in various ﬁelds, such as image superresolution (Shocher et al., 2018) and domain adaption (Sun et al., 2020; Wang et al., 2021).
While our work is thus closely related to Test-Time Training (TTT) in (Sun et al., 2020), we target a signiﬁcantly different scenario.
TTT assumes that all test samples have been subject to the same distribution shift compared with the training data.
As a consequence, it incrementally updates the model parameters when receiving new test images.
By contrast, in our scenario, there is no systematic distribution shift, and it is therefore more effective to ﬁne-tune the parameters of the original model for every new test mini-batch.
This motivates our MAT strategy, which searches for the initial model parameters that can be effectively ﬁne-tuned in a self-supervised manner.
T HEORY OF T EST - TIME A DAPTATION
In this section, we study the relationship between the test-time adaptation and Bayesian optimal robustness, which represents the optimal robustness that can be achieved from the training data, showing that the test-time adaptation can extend the function classes and improve the robustness of the model.
Deﬁnition 3.1.
For a model F ( x ) with (cid:96) 2 adversarial constraint (cid:107) x (cid:63) − x (cid:107) < ε , we deﬁne its natural risk and adversarial risk as at point x
[Equation]
Remarks.
We use the MSE loss to deﬁne the natural risk
R nat x ( F ) and adversarial R adv x ( F ) at point x . Similar to TRADES (Zhang et al., 2019)
The adversarial risk is deﬁned as the sum of natural risk and the loss changes under adversarial attack, and it can be bounded by the maximum MSE loss within the adversarial budget
[Equation]
Therefore, for the adversarial input x (cid:63) with (cid:107) x (cid:63) − x (cid:107) < ε , small R adv x ( F ) guarantee small test error on x (cid:63) .
Small R nat x ( F ) and R adv x ( F ) represents good clean performance and high adversarial robustness respectively
In the following deﬁnitions, we deﬁne three algorithms to obtain adversarially robust functions and compare their adversarial risks.
Deﬁnition 3.2 (Adversarial Training with TRADES) .
We deﬁne ˆ F AT as
[Equation]
where x i represents the i -th clean training data and y i represents the clean training label.
Remark.
Empirically, adversarial training is a very popular method to achieve robustness.
It minimizes the adversarial risk on the training data.
Then we consider the Bayesian optimal robustness.
Let F represent a function class.
We assume the response is generated by y = F ∗ ( x ) + ξ with prior distribution F ∗ ∈ F ∼ P F and ξ ∼ P ξ .
Denote X ∈ R n × d , Y ∈ R n as the training data and training response generated by y i =
F ∗ ( x i ) + ξ .
For problems like Bayesian linear regression (Bishop & Nasrabadi, 2006), function ˆ F ∈ F is able to achieve the Bayesian optimal natural risk
E X , Y ,y ( ˆ F ( x ) − y ) 2 .
However, the function class F is not enough to achieve the Bayesian optimal adversarial risk.
The adversarial risk depends on the local Lipschitz of function ˆ F , in order to better trade-off between the Lipschitz and natural risk of the function F , much more complicated function classes than F are needed to achieve the optimal adversarial robustness.
We deﬁned the two
Bayesian functions F RB and F AB that minimize global adversarial risk and adversarial risk at the speciﬁc point x .
F AB extends the function class beyond F and achieves better robustness.
Deﬁnition 3.3 (Restricted Bayesian Robust Function F RB ) .
The restricted Bayesian robust function
F RB minimizes the global adversarial risk inside the function class F
[Equation]
Remark.
The Bayesian function represents the best robust function inside the function class F For any F ∈ F , no training algorithms can achieve better average adversarial risk than F RB .
Deﬁnition 3.4 (Adaptive Bayesian Robust Function F AB ) .
The adaptive Bayesian robust function
F AB inside the function class F that minimizes the adversarial risk at point x is
[Equation]
Remark.
Instead of minimizing the global average R adv x ( F ) , F AB minimizes the adversarial risk in the given input point x .
The function depends on the input x so that the model extends the function class beyond F .
For different test inputs, we can use different functions to achieve the optimal adversarial risk.
Therefore, we refer to F AB as the test-time adapted function.
In the following theorem, we show the difference between three functions in the model, where the test-adapted function F AB signiﬁcantly improves the robustness.
Theorem 3.1 (Linear Models) .
We consider a linear function classes F Lin =
{ F Lin | F Lin ( x ; θ ) =
x (cid:62) θ , θ ∈ R d } .
The output y is generated by y = x (cid:62) θ ∗ + ξ , where θ ∗ is independent of x with θ ∗ ∼ N (0 , τ 2 I ) , and the noise ξ ∼ N (0 , σ 2 ) .
Let X ∈ R n × d , Y ∈ R n denote the training data and the responses respectively.
For linear model F Lin ( x ; θ ) , three estimators in Deﬁnition 3.2 to 3.4 are
[Equation]
where (cid:98) θ
Linnat =
X ( X (cid:62) X
+ λ ∗ I n ) − 1 Y , λ ∗ =
σ 2 /τ 2 .
Furthermore, if each dimension of x is i.i.d.
with E x = 0 , Cov ( x ) = I
d /d and E [ √ dx i ] 4 ≤ M for some universal constant M < ∞ , denoting ∆ =
(1 + c + λ ∗ ) 2 − 4 c , then when n, d → ∞ with n/d = c ∈ (0 , 1)
R adv x ( (cid:98) θ LinAT ) =
τ 2 ,
[Equation]
And when c → 1 with SNR = σ 2 /τ 2 ≤ 1 , R adv x ( (cid:98) θ LinAB ) <
R adv x ( (cid:98) θ LinRB )(1 − 2 3( ε 2 +1) ) .
Remarks.
In this theorem, we provide the form of the three estimators and their adversarial risks.
The gap of adversarial risk between (cid:98) θ LinAT and (cid:98) θ LinRB vanishes when n, d → ∞ .
The estimator (cid:98) θ LinRB achieves the optimal robust risk among all linear models.
However, for an arbitrary ratio c = n/d , R adv x ( (cid:98) θ LinAB ) <
R adv x ( (cid:98) θ LinRB ) , indicating that adaptation to each test data x can improve the robustness of the model even when compared with the best linear model.
Theorem 3.1 provides the optimal test-time adapted estimator in the linear function classes F Lin , which depends on the clean input x .
In Figure 1, we plot the adversarial risk of three estimators for different adversarial budgets, which clearly shows that our adaptation can signiﬁcantly increase the robustness.
When the input is corrupted with adversarial noise, the same form of the test-time adapted estimator also signiﬁcantly improves the adversarial risk shown in the following theorem.
Theorem 3.2 (Corrupted Input) .
We assume the oracle parameter θ ∗ is independent of x and has the prior distribution θ ∗ ∼ N (0 , τ 2 I ) , and the noise ξ ∼ N (0 , σ 2 ) .
Furthermore, each dimension of x is i.i.d.
with E x = 0 , Cov ( x ) = I
d /d and E [ √ dx i ] 4 ≤ M for some universal constant M < ∞ , then when n, d → ∞ with n/d = c ∈ (0 , 1) .
Given corrupted input x (cid:63) =
x + ε (cid:98) θ / (cid:107) (cid:98) θ (cid:107) , with ε < 1 , the adversarial risk of (cid:98) θ LinAB ,(cid:63) =
( x (cid:63) x (cid:63) (cid:62) + ε 2 I d ) − 1 x (cid:63) x
(cid:63) (cid:62) X
( X (cid:62) X
+ λ ∗ I n ) − 1 Y is
R adv x ( (cid:98) θ Lin AB ,(cid:63) ) =
[Equation]
[Figure]
Remarks.
The theorem shows that when the given input is adversarial, the test-time adaptation can still lower the adversarial risk of the model as R adv x ( (cid:98) θ LinAB ) <
R adv x ( (cid:98) θ LinAB ,(cid:63) ) <
R adv x ( (cid:98) θ LinRB ) .
In the statistical Bayesian model, we show that the testtime adaptation can extend the function classes and achieve the signiﬁcantly lower adversarial risk than the ﬁxed model.
In the practical non-Bayesian classiﬁcation task, explicit calculation of the optimal model is difﬁcult.
Nevertheless, the test-time adaptation also helps to improve the robustness of the model.
As will be shown in the following section, we perform the self-supervised test-time ﬁne-tuning to adapt the model to each input, and largely improves the robust accuracy of the test-time adapted model.
M ETHODOLOGY
We follow the traditional multitask learning formulation (Caruana, 1997) and consider a neural network with a backbone z = E ( x ; θ E ) and K + 1 heads.
One head f ( z ; θ f ) outputs the classiﬁcation result while the other K heads g 1 ( z ; θ g 1 ) , ..., g K ( z ; θ gK ) correspond to K auxiliary self-supervised tasks.
θ =
( θ E , θ f , θ g 1 , · · · , θ gK ) encompasses all trainable parameters, and we further deﬁne
[Equation]
Furthermore, let D = { ( x i , y i ) } ni =1 denote the training set, and (cid:101)
D
= { ( (cid:101) x i , (cid:101) y i ) } mi =1 be the test set.
For further illustration, the labels of the test set are shown.
However, they are unknown to the networks at test time.
We denote the adversarial examples of x as x (cid:63) .
It satisﬁes (cid:107) x
(cid:63) − x (cid:107) ≤ ε , and ε is the size of the adversarial budget.
For any set S , we represent its average loss as
[Equation]
where | S | is the number of elements in S .
The general classiﬁcation loss, such as the cross-entropy, is denoted by L cls .
We use the superscript “AT” to denote the adversarial training loss.
For example,
[Equation]
S ELF -S UPERVISED T
EST - TIME F INE - TUNING
Our goal is to perform self-supervised learning on the test examples to mitigate the overﬁtting problem of AT and adapt the model for each data point.
To this end, let us suppose that an adversarially-trained network with parameters θ 0 receives a mini-batch of b adversarial test examples (cid:101) B (cid:63)
= { ( (cid:101) x
(cid:63) 1 , (cid:101) y 1 ) , · · · , ( (cid:101) x (cid:63)b , (cid:101) y b ) } , As the labels { (cid:101)
y i } bi =1 are not available, we propose to ﬁne-tune the backbone parameters θ E by optimizing the loss function
[Equation]
which encompasses K self-supervised tasks.
Here, L SS,k represents the loss function of the k -th task and { C k } Kk =1 are the weights balancing the contribution of each task.
In our experiments, the L
SS,K is the cross-entropy loss to predict rotation and vertical ﬂip.
The number of images b may vary from 1 to m .
b = 1 corresponds to the online setting, where only one adversarial image is available at a time, and the backbone parameters θ E are adapted to every new image.
The online setting is the most practical one, as it does not make any assumptions about the number of adversarial test images the network receives.
By contrast, b = m corresponds to the ofﬂine setting, where all adversarial test examples are available at once.
It is similar to transductive learning (Gammerman et al., 1998; Vapnik, 2013).
Note that our online setting differs from the online test-time training described in TTT (Sun et al., 2020); we do not incrementally update the network parameters as new samples come, but instead initialize ﬁne-tuning from the same starting point θ 0 for each new test image.
Eqn (4) encourages θ E to update in favor of the self-supervised tasks.
However, as the classiﬁcation head f was only optimized for the old backbone E ( · ; θ 0 E ) , it will typically be ill-adapted to the new parameters θ ∗ E , resulting in a degraded robust accuracy.
Furthermore, for a small b , the model tends to overﬁt to the test data, reducing L SS to 0 but extracting features that are only useful for the self-supervised tasks.
To overcome these problems, we add an additional loss function acting on the training data that both regularizes the backbone E and optimizes the classiﬁcation head f
so that f remains adapted to the ﬁne-tuned backbone E ( · ; θ ∗ E ) .
Speciﬁcally, let B ⊂ D denote a subset of the training set.
We then add the regularizer
[Equation]
(5) to the ﬁne-tuning process.
In short, Eqn (5) evaluates the AT loss on the training set to ﬁne-tune the parameters θ f of the classiﬁcation head.
It also forces the backbone E to extract features that can be used to make correct predictions, i.e., to prevent θ E from being misled by L SS when b is small.
Combining Eqn (4) and Eqn (5), our ﬁnal test-time adaptation loss is
[Equation]
where C sets the inﬂuence of L R .
The algorithms that describe our test-time self-supervised learning are deferred to Appendix D. As SGD is more efﬁcient for larger amount of data, we use SGD to optimize θ when b is large ( e.g. ofﬂine setting).
4. M ETA A DVERSARIAL T RAINING
To make the best out of optimizing L test at test time, we should ﬁnd a suitable starting point θ 0 , i.e., a starting point such that test-time self-supervised learning yields better robust accuracy.
We translate this into a meta learning scheme, which entails a bilevel optimization problem.
Speciﬁcally, we divide the training data into s small exclusive subsets D = ∪ sj =1 B j and let B (cid:63)j to be adversaries of B j .
We then formulate meta adversarial learning as the bilevel minimization of
L meta ( D ; θ ) =
1 s (cid:88)
[Equation]
L
[Equation]
, where
[Equation]
arg
[Equation]
where L SS is the self-supervised loss function deﬁned in Eqn (4) and L ATcls is the loss function of
AT deﬁned in Eqn (3).
As bilevel optimization is time-consuming, following MAML (Finn et al., 2017), we use a single gradient step of the current model parameters θ to approximate θ ∗ j .
[Equation]
In essence, this Meta Adversarial Training (MAT) scheme searches for a starting point such that ﬁne-tuning with L SS will lead to good robust accuracy.
If this holds for all training subsets, then we can expect the robust accuracy after ﬁne-tuning at test time also to increase.
Note that, because the meta learning objective of Eqn (7) already accounts for classiﬁcation accuracy, the regularization by L
R is not needed during meta adversarial learning.
Accelerating Training.
To compute the gradient ∇ θ L meta ( D ; θ ) , we need to calculate the timeconsuming second order derivatives − α ∇ 2 θ L SS ( B (cid:63)j ; θ ) ∇ θ ∗ j L ATcls ( B j ; θ ∗ j ) .
Considering that AT is already much slower than standard training (Shafahi et al., 2019), we cannot afford another significant training overhead.
Fortunately, as shown in (Finn et al., 2017), second order derivatives have little inﬂuence on the performance of MAML.
We therefore ignore them and take the gradient to be
[Equation]
However, by ignoring the second order gradient, only the parameters on the forward path of the classiﬁer F , i.e. , θ E and θ f , will be updated.
In other words, optimizing Eqn (7) in this fashion will not update { θ gk } Kk =1 .
To nonetheless encourage each self-supervised head G k to output the correct prediction, we incorporate an additional loss function encoding the self-supervised tasks,
[Equation] [Equation]
Note that we use the adversarial version of L SS to provide robustness to the self-supervised tasks, which, as shown in (Chen et al., 2020a; Hendrycks et al., 2019; Yang & Vondrick, 2020), is beneﬁcial for the classiﬁer.
The ﬁnal meta adversarial learning objective therefore is
[Equation]
(11) where C (cid:48) balances the two losses.
Algorithm 1 shows the complete MAT algorithm.
E XPERIMENTS
Experimental Settings.
Following previous works (Cui et al., 2020; Huang et al., 2020), we consider (cid:96) ∞ -norm attacks with an adversarial budget ε = 0 .
031 ( ≈ 8 / 255 ).
We evaluate our method
Input: Training set D ; Learning rate α, β ; Iterations T ; Weights C k and C (cid:48)
Output: Starting parameters θ 0 for the test-time ﬁne-tuning 1: for t = 1 to T do 2:
Sample q exclusive batches of training images B 1 , B 2 , · · · , B q ⊂ D 3:
Using PGD to ﬁnd the adversaries B (cid:63)j : x (cid:63)j,i = arg max (cid:107) x (cid:63)j,i − x j,i (cid:107)≤ ε L cls ( F ( x (cid:63)j,i ) , y j,i ) 4: for batches B 1 , B 2 , · · · , B q do 5: θ ∗ j = θ − α ∇ θ L SS ( B (cid:63)j ; θ ) 6: l meta,j = L ATcls ( B j ; θ ∗ j ) 7: end for 8: θ = θ − βq (cid:80) B j (cid:104) ∇ θ ∗ j l meta,j + C (cid:48) ∇ θ L ATSS ( B j ; θ ) (cid:105) 9: end for 10: return Trained parameters θ
= θ on three datasets: CIFAR10 (Krizhevsky et al., 2009), STL10 (Coates et al., 2011) and Tiny ImageNet (Le & Yang, 2015).
We also use two different network architectures: WideResNet-3410 (Zagoruyko & Komodakis, 2016) for CIFAR10, and ResNet18 (He et al., 2016) for STL10 and
Tiny ImageNet.
The hyperparameters are provided in the Appendix D.
Self-Supervised Tasks.
In principle, any self-supervised tasks can be used for test-time ﬁne-tuning, as long as they are positively correlated with the robust accuracy.
However, for the test-time ﬁnetuning to remain efﬁcient, we should not use too many self-supervised tasks.
Furthermore, as we aim to support the fully online setting, where only one image is available at a time, we cannot incorporate a contrastive loss (Chen et al., 2020b;
He et al., 2020; Kim et al., 2020) to L SS .
In our experiments, we therefore use two self-supervised tasks that have been shown to be useful to improve the classiﬁcation accuracy: Rotation Prediction and Vertical Flip Prediction .
Attack Methods.
In the white-box attacks, the attacker knows every detail of the defense method.
Therefore, we need to assume that the attacker is aware of our test-time adaptation method and will adjust its strategy for generating adversarial examples accordingly.
Suppose that the attacker is fully aware of the hyperparameters for test-time adaptation.
Then, ﬁnding adversaries (cid:101) B (cid:63) of the clean subset (cid:101) B can be achieved by maximizing the adaptive loss
[Equation]
where L attack refers to the general attack loss, such as the cross-entropy or the difference of logit ratio (DLR) (Croce & Hein, 2020a).
We call this objective in Eqn (12) adaptive attack , which can be either performed in white-box or black-box attacks.
We consider four common white-box and black-box attack methods: PGD-20 (Madry et al., 2018), AutoPGD (both cross-entropy and DLR loss) loss (Croce & Hein, 2020a), FAB (Croce & Hein, 2020b) and Square Attack (Andriushchenko et al., 2020).
We apply both the standard and adaptive versions of these methods.
Particularly,
AutoPGD we use is a strong version that maximizes the loss function that continues when ﬁnding adversarial examples (Croce et al., 2022).
More details are provided in the Appendix E.
Baselines.
We compare our method with the following methods: 1) Regular AT, which uses L ATcls in Eqn (3).
Regular AT with an additional self-supervised loss, i.e., using L ATcls + C (cid:48)
L ATSS for AT, where L ATSS is given in Eqn (10).
This corresponds to the formulation of (Hendrycks et al., 2019).
3) MAT (Algorithm 1) without test-time ﬁne-tuning.
5. R OBUST A CCURACY
CIFAR10.
Table 1a shows the robust accuracy for different attacks and using two different tasks for ﬁne-tuning.
The adaptive attack is not applicable to models without ﬁne-tuning.
As we inject different self-supervised tasks into the AT stage, and as different self-supervised tasks may impact the robust accuracy differently (Chen et al., 2020a), the robust accuracy without ﬁne-tuning still varies.
The vertical ﬂipping task yields better robust accuracy before ﬁne-tuning but its improvement after ﬁne-tuning is small.
By contrast, rotation prediction achieves low robust accuracy before ﬁne-tuning,
[Table] [Table]
Tasks E ( ρ ( (cid:101) x (cid:63) ))
P ( ρ ( (cid:101) x (cid:63) ) > 0)
Rotation 0.
VFlip
[Figure]
[Figure]
but its improvement after ﬁne-tuning is the largest.
Using both tasks together combines their effect and yields the highest overall accuracy after test-time adaptation.
Note that our self-supervised testtime ﬁne-tuning, together with meta adversarial learning, consistently improves the robust accuracy under different attack methods.
Under the strongest adaptive AutoPGD, test-time ﬁne-tuning using both tasks achieves a robust accuracy of 57.70%, signiﬁcantly outperforming regular AT.
STL10 and Tiny ImageNet.
As using both the rotation and vertical ﬂip prediction led to the highest overall accuracy on CIFAR10, we focus on this strategy for STL10 and Tiny ImageNet.
Table 1b and 1c shows the robust accuracy on STL10 and Tiny ImageNet using a ResNet18.
Our approach also signiﬁcantly outperforms regular AT on these datasets.
Ofﬂine Test-time Adapattion.
As shown in Table 1a, 1b, 1c, the ofﬂine ﬁne-tuning further improves the robust accuracy over the online version.
Diverse Attacks.
Recommended by (Croce et al., 2022), in Appendix C.1, we evaluate our method on diverse attacks including transfer attack , expectation attack and boundary attack , where test-time adaptation all improves the robustness of the model.
5. M ETHOD A NALYSIS
We observe the signiﬁcant positive correlation between the gradient of self-supervised loss L SS and the classiﬁcation loss L cls .
Deﬁne
[Equation]
) and approximate L cls by the Taylor expansion
[Equation]
As θ E contains millions of parameters, its gradient norm is typically large.
Therefore, gradient descent w.r.t.
L SS should act as a good substitute for optimizing L cls when ρ ( (cid:101)
x
(cid:63)i ) is signiﬁcantly larger than 0.
We further conﬁrm this empirically.
For all adversarial test inputs (cid:101) x (cid:63) ∼ (cid:101) D (cid:63) , we regard ρ ( (cid:101) x
(cid:63) ) as a random variable and calculate its empirical statistics on the test set.
Table 2 shows the empirical statistics of an adversarially-trained model on CIFAR10, and Figure 2 shows the c.d.f.
of ρ ( (cid:101) x (cid:63) ) .
The mean of ρ ( (cid:101) x (cid:63) ) is indeed signiﬁcantly larger than 0 and P ( ρ ( (cid:101)
x (cid:63) ) > 0) is larger than the robust accuracy of the adversarially-trained network (50%-60%), which implies that self-supervised test-time ﬁne-tuning helps to correctly classify the adversarial test images.
We further provide the theoretical analysis in a linear model in Theorem B.1, which shows that the correlated gradient signiﬁcantly strengthens the robustness and lowers natural risk.
Besides, the correlated gradient also helps the model to move closer to the Bayesian robust estimator (cid:98) θ AB .
5. A BLATION S TUDY
Meta Adversarial Training.
To show the effectiveness of MAT, we perform an ablation study to ﬁne-tune the model with regular AT ( i.e. , setting α = 0 in line 5 of Algorithm 1).
Table 7 shows that the robust accuracy and the improvements of ﬁne-tuning are consistently worse without MAT.
Accuracy Improvement on Inputs with Different Adversarial Budget.
As shown in Table 8, we set ε = 0 .
015 to perform the online test-time ﬁne-tuning, showing that our method is also able to improve the robust accuracy of inputs with different adversarial budgets.
Removing L SS or L R .
To study the effect of L SS and L R in L test , we report the robust accuracy after online ﬁne-tuning using only L R and L SS in Table 9.
While, as expected, removing L SS tends to reduce more accuracy than removing L R .
It shows the beneﬁts of our self-supervised test-time ﬁne-tuning strategy.
Nevertheless, the best results are obtained by exploiting both loss terms.
Improvement on Clean Images.
As predicted by Theorem 3.1 and B.1, our method is able to improve not only the robust accuracy but also the natural accuracy.
As shown in Table 10, our approach increases the clean image accuracy by test-time adaptation.
This phenomenon further strengthens our conjecture that the improvement of robust accuracy is due to the improvement of generalization instead of gradient masking.
C ONCLUSION
In linear models and two-layer random networks, we theoretically demonstrate the necessity of testtime adaptation for the model to achieve optimal robustness.
To this end, we propose self-supervised test-time ﬁne-tuning on adversarially-trained models to improve their generalization ability.
Furthermore, we introduce a MAT strategy to ﬁnd a good starting point for our self-supervised ﬁne-tuning process.
Our extensive experiments on CIFAR10, STL10 and Tiny ImageNet demonstrate that our method consistently improves the robust accuracy under different attack strategies, including strong adaptive attacks where the attacker is aware of our test-time adaptation technique.
In these experiments, we utilize three different sources of self-supervision: rotation prediction, vertical ﬂip prediction and the ensemble of them.
A P ROOFS OF T HEOREMS A. P RELIMINARY : M ARCHENKO -P ASTUR L AW AND T RANSFORMATION OF E IGENVALUES
Before the proof linear models, we ﬁrst give the asymptotic spectrum of the matrix K
= X (cid:62) X , and some useful results of the trace of the transformation of K .
Lemma A.1 (Marchenko-Pastur Law, Theorem 3.4 in (Bai & Silverstein, 2010)) .
We deﬁne the eigenvalues of X (cid:62) X : λ 1 > λ 2 > · · · λ n has distribution with c.d.f.
[Equation]
If each dimension of x is i.i.d.
with E x = 0 , Cov ( x ) = I
d /d and E [ √ dx i ] 4 ≤ M for some universal constant M < ∞ , then when n, d → ∞ with n/d = c ∈ (0 , 1) , for any bounded function g , when n, d → ∞ with c = n/d ∈ [0 , ∞ )
[Equation]
with p.d.f.
dH ( s )
[Equation]
where λ − = (1 − √ c ) 2 and λ + = (1 + √ c ) 2 .
Lemma A.2.
If each dimension of x is i.i.d.
with E x = 0 , Cov ( x ) = I
d /d and E [ √ dx i ] 4 ≤ M for some universal constant M < ∞ , then when n, d → ∞ with n/d = c ∈ (0 , 1) ,
[Equation]
where ∆ =
(1 + c + z ) 2 − 4 c .
Proof.
We deﬁne three transformations of the eigenvalue of dH ( s ) t
[Equation]
for z ∈ [0 , ∞ )
They can be calculated by the Stieltjes transformation of Marchenko-Pastur law.
By Marchenko-Pastur semicircular law, the Stieltjes transformation of dH ( s ) is (Lemma 3.11 in (Bai & Silverstein, 2010) and Lemma 4.4 in (Cheng et al., 2022))
[Equation]
Let ∆ =
(1 + c + z ) 2 −
[Equation]
c
[Equation]
, and
[Equation]
Then
[Equation]
The trace operation can be translated into:
[Equation]
Therefore, when n
[Equation]
d → ∞ with c = n/d ∈ (0 , 1) ,
[Equation]
where ∆ =
(1 + c + λ ) 2 − 4 c .
A. P ROOF OF T HEOREM 3.
We separate the proof into three parts for three estimators separately.
Proof of (cid:98) θ LinAT .
For (cid:98) θ ATLin , taking gradient with respect to the objective function
[Equation]
we obtain
[Equation]
Therefore
[Equation]
Its natural risk is
[Equation] [Equation]
[Equation]
When λ = nε 2 → ∞ , by Lemma A.2,
[Equation]
Therefore,
[Equation]
Lemma A.3.
If the oracle parameter θ ∗ is independent of x and has the prior distribution θ ∗ ∼ N (0 , τ 2 I ) , and the noise ξ ∼ N (0 , σ 2 ) .
Furthermore, if we assume each dimension of x is i.i.d.
with E x = 0 , Cov ( x ) = I
d /d and E [ √ dx i ] 4 ≤ M for some universal constant M < ∞ , when n, d → ∞ with n/d = c ∈ (0 , 1) , then for λ ∗ =
σ 2 /τ 2 and (cid:98) θ = 1 A X ( X (cid:62) X + λ ∗ I n ) Y ,
[Equation]
where ∆ =
(1 + c + λ ∗ ) 2 − 4 c .
Proof.
[Equation]
According to Lemma A.2, when n, d → ∞ with c = nd ∈ (0 , 1) ,
[Equation]
where λ ∗ =
σ 2 /τ 2 and ∆ =
(1 + c + λ ∗ ) 2 − 4 c .
Therefore,
[Equation]
linbayes )
=
τ 2 − τ 2 A ((1 + c + λ
[Equation]
) − √ ∆
[Equation]
(1 − 1 2 A ) .
And
[Equation]
Therefore,
[Equation]
Proof of (cid:98) θ LinRB .
It is well known that the posterior distribution of θ ∗ is
[Equation]
where λ ∗ =
σ 2 /τ 2 .
According to the deﬁnition of Bayesian estimator
[Equation]
As the linear model only allows ﬁxed (cid:98) θ for each x
[Equation]
we obtain
[Equation]
Therefore,
[Equation]
Using Lemma A.
[Equation]
when d → ∞
[Equation]
Summarizing the results,
[Equation]
Proof of (cid:98) θ LinAB .
Bayesian robust estimator of each x , which optimizes ( x (cid:62) ( (cid:98) θ − θ ∗ )) 2 + ε 2 (cid:107) (cid:98) θ (cid:107) 2 is
[Equation]
where it is well known that the posterior distribution of θ ∗ is
[Equation]
Taking the gradient w
[Equation]
r
[Equation]
t (cid:98) θ gives the solution
[Equation] [Equation]
where (cid:98) θ
Linnat =
X
[Equation]
) with λ ∗ =
σ 2 /τ 2 is the Bayesian estimator for natural risk.
For its natural risk R nat x (
[Equation]
Lin
[Equation] [Equation]
As
[Equation]
then
[Equation]
When d → ∞ , x (cid:62) x → 1 in probability.
Therefore,
[Equation]
Then when d → ∞ ,
[Equation]
By Lemma A.3,
[Equation]
As x (cid:62) x
→ 1 in probability when d → ∞ ,
[Equation] [Equation]
2 +1) 2 .
Then
[Equation] [Equation]
From Lemma A.3,
[Equation]
Summarizing two parts, the adversarial risk is
[Equation]
A. P ROOF OF T HEOREM 3.
Proof.
As (cid:98) θ LinAB
=
xx (cid:62) (cid:98) θ
Linnat ε 2 + x (cid:62) x , the adversarial input
[Equation]
As d → ∞ , (cid:107) x (cid:107) → 1 in probability.
Therefore, x (cid:63) = (1 + ε ) x .
Taking it into (cid:98) θ LinAB ,(cid:63) =
[Equation]
x
[Equation]
x
[Equation]
I
[Equation]
, we obtain (cid:98) θ LinAB ,(cid:63)
[Equation]
Linnat
[Equation]
+
[Equation]
x
From Eqn (30) and 31,
R nat x ( (cid:98) θ LinAB ,(cid:63) ) =
τ(cid:18) 1 − (1 + ε )ε
+ (1 + ε ) 2 (1 + c + λ ∗ − √ ∆) (cid:19)
+ τ 2 (1 + ε )2( ε
+
(1 + ε ) 2 ) 2 (1 + c + λ ∗ − √ ∆)
L ( (cid:98) θ LinAB ,(cid:63) )
= τ 2 (1 + ε )2( ε 2 + (1 + ε ) 2 )
(1 + c + λ ∗ − √ ∆) .
Therefore,
R adv x ( (cid:98) θ LinAB ,(cid:63) )
=
R nat x ( (cid:98) θ LinAB ,(cid:63) ) + ε 2 L ( (cid:98) θ LinAB ,(cid:63) ) 2 = τ(cid:32)
[Equation]
B C ORRELATED G RADIENTS
In the following Theorem, we show that with correlated gradient, one gradient descent step like our method largely improves the natural and adversarial risk of the model.
Theorem B.1.
We assume the oracle parameter θ ∗ is independent of x and has the prior distribution θ ∗ ∼ N (0 , τ 2 I ) , and the noise ξ ∼ N (0 , σ 2 ) .
Let (cid:98) θ
= (cid:98) θ
AT be the estimator of adversarial training of the linear model F Lin ( x ; θ ) :
[Equation]
When receiving a new test data point ( x (cid:63) , y ) and taking one gradient descent step with correlated gradient (cid:98) g : (cid:98) θ 1 = (cid:98) θ 0 − η ˆ g , where (cid:107) x
(cid:63) − x (cid:107) ≤ ε is an adversarial example of x ,
Corr (ˆ g , ∇ (cid:98) θ 0 L ( F Lin ( x , (cid:98) θ 0 ) , y ))
= ρ > 0 and η is the learning rate.
Let x = ( x 1 , · · · , x d ) where x i is the i -th element of x .
We further assume that { x i } di =1 are i.i.d.
with E [ x i ] = 0 , Var [ x i ] = 1 /d .
And E [ √ dx i ] 4 ≤ M for some universal constant M < ∞ .
When n, d → ∞ with c = n/d ∈ (0 , 1) , with the optimal learning rate,
[Equation]
Remarks.
With correlated gradients, improvements of R nat x and R adv x are both positive when having ρ > 0 .
By taking correlated gradient descent on the parameter, we get large improvements of both natural and adversarial risks.
Theorem B.1 shows that ﬁne-tuning with correlated gradient largely improves both clean performance and robustness of the models.
In addition, for linear models, the Bayesian optimal estimator is (cid:98) θ LinAB (cid:107) x .
And (cid:98) θ 1 (cid:107) ˆ g with Corr ( x (cid:63) , ˆ g )
=
ρ .
As x (cid:63) is close to x , with a proper learning rate, we can get close to Bayesian robust estimator with correlated gradient descent.
B. P ROOF OF T HEOREM B.
Proof.
For a new test input ( x (cid:63) , y ) , where (cid:107) x
(cid:63) − x (cid:107) ≤ ε is an adversarial example near the input, its MSE loss is
[Equation]
Taking the gradient w.r.t θ
[Equation]
Suppose the self-supervised task gives a correlated version of gradient and updates (cid:98) θ 0 with one step of gradient descent
[Equation]
where Corr (ˆ g , x (cid:63) )
=
ρ and E [ˆ g (cid:62) ˆ g ]
=
E [ x (cid:63) (cid:62) x (cid:63) ] .
From Theorem 3.1, when λ = nε 2 → ∞ ,
R nat x ( (cid:98) θ 0 ) and (cid:107) (cid:98) θ
(cid:107) 2 can be simpliﬁed as:
[Equation]
Therefore, when d → ∞
[Equation]
Then when d → ∞ ,
[Equation]
Therefore,
E
[Equation] [Equation]
[Equation]
By decomposing ˆ g = ρ x (cid:63) + (cid:112) 1 − ρ 2 z ∗ , we can obtain,
[Equation]
x ,
[Equation]
As (cid:107) x
[Equation]
−
[Equation]
ε ,
[Equation]
For natural risk
[Equation]
By (cid:107) x (cid:63) − x (cid:107) ≤ ε , Corr (ˆ g , x (cid:63) )
=
ρ and E [ˆ g (cid:62) ˆ g ]
=
E [ x
[Equation]
Therefore,
[Equation]
Optimizing η get
[Equation]
With η ∗ ,
[Equation]
) and
R nat x ( (cid:98)
[Equation]
Compared with (cid:98) θ
[Equation]
we have improvements of
[Equation]
and
[Equation]
C A DDITIONAL E XPERIMENTS C. D IVERSE A TTACKS
Transfer Attack.
In Table 3, we perform a transfer attack from the static adversarial defense.
We use the robust networks with the same architecture as the substitute model, and the test-time adaptation also improves the robust accuracy.
Expectation Attack.
In Table 4, we show the results of the expectation attack.
We modify the adaptive attack and average the gradient from 10 ﬁne-tuned models, whose training batches are different.
We evaluate the model using the ensemble of rotation and vertical ﬂip as the self-supervised task on
CIFAR10.
We evaluate the model with Adaptive-AutoPGD-EOT and Adaptive-SquareAttack-EOT.
One is the strongest attack in our method and the other is a black-box attack that is less likely to be
[Table]
[Table]
affected by gradient masking.
The experiment shows that the expectation attack has little inﬂuence on the improvement of our test-time adaptation.
Boundary Attack.
We use one of the SOTA decision-based attacks: RayS (Chen & Gu, 2020).
We test it on CIFAR10 with the ensemble of rotation and vertical ﬂip.
Table 5 shows that our method also improves the robust accuracy of the decision-based attack.
GMSA (Chen et al., 2021a) with AutoPGD.
GMSA is a recently proposed attack algorithm targeted at the test-time model adaptation.
We use the GMSA with AutoPGD to attack our method, and the results are shown in Table 6.
Under GMSA, our test-time adaptation still signiﬁcantly improves the robust accuracy.
Moreover, Table 6 also demonstrates the strength of our adaptive attack strategy as it achieves a higher success rate than GMSA.
C. A BLATION S TUDY
Meta Adversarial Training.
Our meta training strategy in Algorithm 1 aims to strengthen the correlation between the self-supervised tasks and classiﬁcation.
To show its effectiveness, we perform an ablation study where we ﬁne-tune the model with regular AT ( i.e. , setting α = 0 in line 5 of
Algorithm 1).
We then perform the same test-time ﬁne-tuning on the model without MAT, using the same hyperparameters as in the MAT case.
As shown in Table 7, the robust accuracy and the improvements of ﬁne-tuning are consistently worse without MAT.
Accuracy Improvement on Inputs with Different Adversarial Budget.
Our method is also able to improve the robust accuracy of inputs with different adversarial budgets.
As shown in Table 8, we set (cid:96) ∞ budget of the adversarial inputs to be 0 .
015 to perform the online test-time ﬁne-tuning.
The robust accuracy is ed improved.
Removing L SS or L R .
In our previous experiments, test-time ﬁne-tuning was achieved using a combination of two loss functions: L SS and L R .
To study the effect of each of these terms separately, we remove either one of them from L test .
In Table 9, we report the robust accuracy after online ﬁne-tuning using only L R and only L SS .
While, as expected, removing L SS tends to reduce more accuracy than removing L R .
It shows the beneﬁts of our self-supervised test-time ﬁne-tuning strategy.
Nevertheless, the best results are obtained by exploiting both loss terms.
Accuracy Improvement on Clean Images.
As shown in Eqn (30) and Theorem B.1, our method is able to improve not only the robust accuracy but also the natural accuracy of clean images on adversarially-trained models.
To evidence this, we maintain all the components of our model and simply replace the adversarial input images with clean images ( i.e. replacing (cid:101) B (cid:63) with clean inputs (cid:101) B in Algorithm 2) and perform the same self-supervised test-time ﬁne-tuning.
As shown in Table 10, our approach increases the clean image accuracy.
This phenomenon further strengthens our conjecture that the improvement of robust accuracy is due to the improvement of generalization instead of perturbing the model parameters, because randomly perturbing the parameters usually lowers the natural accuracy of the model.
[Table]
[Table]
Attacking Objectives.
The improvement of the test-time adaptation is not affected by the attack objectives.
Even if no information of the ground truth label is incorporated in the attack, the testtime adaptation improves the robust accuracy.
When the attacker randomly lowers the score of the false label to perform the adversarial attack, if our method uses the information of the leaked label to improve the robust accuracy, it will predict the false label and reduce the accuracy.
However, as shown in Table 11, the self-supervised test-time ﬁne-tuning improves the robust accuracy on these “adversarial” images.
Besides, previous experiments on clean images already show that test-time ﬁne-tuning is effective even if there is no information of the ground truth label.
C. A DDITIONAL C OMPARISON
Comparison with SOAP (Shi et al., 2020).
Our method is different from SOAP as we are ﬁnetuning the model to adapt to new examples instead of purifying the input.
We apply SOAP-RP to the adversarially-trained model and ﬁnd that its improvement is marginal.
Under AutoPGD, the accuracy is improved from 53.09% to 53.57%.
This improvement is much smaller than our method, whose improvement is from 53.09% to 57.93%.
SOAP only has little effect when combined with the commonly used AT.
Combination with (Gowal et al., 2020).
We combine our test-time adaptation with AT using additional data (Gowal et al., 2020).
We apply our Meta AT to it with the ensemble of rotation and vertical ﬂip.
Using a WideResNet-28-10, it achieves a robust accuracy of 62.07% under AutoPGD.
With our test-time adaptation, the robust accuracy is improved to 64.34%.
The improvement of robust accuracy is 2.27%.
Robust Accuracy v.s Fine-tuning Steps.
Figure 3 shows the robust accuracy at each step of the test-time ﬁne-tuning for different self-supervised tasks and attack methods.
When using the standard version of attacks, the robust accuracy gradually increases as ﬁne-tuning proceeds.
When using our adaptive attacks, the adversarial examples are generated to attack the network with θ T ( T = 10 ) instead of θ 0 .
Thus, when the parameters gradually change from θ 0 to θ T , the accuracy drops.
Inference Time.
Table 12 shows the inference time for different methods.
While the inference time for our method is larger than SOAP and the normal method when the batch size is 1, the inference time gets closer when using a larger batch size.
And the batch size of 20 or more is a common scenario of the inference.
In order to achieve the statistical optimal adversarial risk, additional time
[Table] [Table]
[Table] [Table]
of test-time adaptation
is necessary.
Reducing the inference time is an important future work of these kinds of methods.
Combination with TRADES (Zhang et al., 2019).
Table 13 shows the robust accuracy of combining our test-time adaptation with TRADES.
Our test-time adaptation improves the robust accuracy by about 4%, which shows our approach can improve various types of robust training methods.
C. V ISUALIZATION
In Figure 4, we show the visualization of several examples that our test-time adaptation successfully corrects the misclassiﬁed examples.
The input examples are generated by AutoPGD on CIFAR10, and we ﬁne-tune the network with the ensemble of Rotation and VFlip tasks.
It shows our testtime adaptation reduces the loss for the whole neighbourhood of the input examples to increase the accuracy of the model.
In Figure 5, we show the histograms of the loss values for the successful and unsuccessful test-time adapted models.
For each input instance, if the test-time adaptation corrects the wrong prediction, we count it as successful.
And if the misclassiﬁed instance is not correctly predicted after our testtime adaptation, it is counted as an unsuccessful one.
The ﬁgure illustrates that our method can adapt the model to correctly classify the instances close to the decision boundary (with medium loss value).
However, for the highly misclassiﬁed instances (with large loss value), which are far away from the decision boundary, our test-time adaptation cannot make the model change so much to predict correct labels for them.
D D ETAILS OF OUR E XPERIMENTAL S ETTING D. H YPERPARAMETERS
Meta Adversarial Training.
The algorithm of Meta Adversarial Training is shown in Algorithm 1.
We consider an (cid:96) ∞ norm with an adversarial budget ε = 0 .
We also use two different network architectures: WideResNet-34-10 for CIFAR10 and ResNet18 for STL10 and Tiny ImageNet.
Following the common settings for AT, we train the network for 100 epochs using SGD with a
[Table]
[Figure]
[Table]
momentum factor of 0.9 and a weight decay factor of 5 × 10 − 4 .
The learning rate β starts at 0.
and is divided by a factor of 10 after the 50-th and again after the 75-th epochs.
The step size α in Eqn (8) is the same as β .
The factor C (cid:48) in Eqn (11) is set to 1 . 0 .
We use 10-iteration PGD (PGD-10) with a step size of 0.007 to ﬁnd the adversarial image B (cid:63)j at training time.
The weight of each self-supervised task is set to C k = 1 K .
We set | B j | = 32 and sample 8 batches B 1 , ..., Bin each iteration.
Furthermore, we save the model after the 51 -st epoch for further evaluation, as the model obtained right after the ﬁrst learning rate decay usually yields the best performance (Rice et al., 2020).
We use PGD with the standard cross-entropy loss to generate adversarial examples at training time in line 3, line 6 and line 8 of Algorithm 1.
The hyperparameters of the attacks are as follows:
• Line 3: PGD-10 with step size 0 .
• Line 6: As θ ∗ j is similar to θ , the adversarial examples at this step are similar to those at Line 4.
To save training time, we therefore choose the starting point of the attack as the adversarial examples in Line 4 and use PGD-2 with a step size of 0 .
• Line 8: PGD-3 with step size 0 .
Online Test-time Fine-tuning.
The algorithm for online ﬁne-tuning is shown in Algorithm 2.
We ﬁne-tune the network for T = 10 steps with a momentum of 0.9 and a learning rate of η = 5 × 10 − 4 .
We set C k = 1 K and C = 15 . 0 .
In line 2 of Algorithm 2, we sample a batch B ⊂ D containingtraining images.
In line 3, we use PGD-10 with a step size of 0 .
Ofﬂine Test-time Fine-tuning.
The algorithm for ofﬂine ﬁne-tuning is shown in Algorithm 3.
As stochastic gradient descent is more efﬁcient for a large amount of data, we use stochastic gradient descent in the ofﬂine ﬁne-tuning.
This is the main difference between Algorithm 2 (online ﬁne-
[Figure]
i
[Figure]
c t i
[Figure]
ﬁne tuned
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
tuning) and Algorithm 3 (ofﬂine ﬁne-tuning).
We also ﬁne-tune the network for 10 epochs.
The batch size of each (cid:101) B (cid:63)j is 128.
The are the online version.
do
Calculate
[Equation]
=
[Equation]
− 1 L test ( (cid:101) B (cid:63) , B ; θ t − 1 ) 6: end for 7: return Prediction (cid:98) y i = arg max j F
( (cid:101) x
(cid:63)i ; θ T )
j
Attacks.
The detailed settings of each attack are provided below:
• PGD-20.
We use 20 iterations of PGD with step size γ = 0 .
The attack loss is the cross-entropy.
• AutoPGD.
We use both the cross-entropy and the difference of logits ratio (DLR) as the attack loss.
The hyperparameters are the same as in (Croce & Hein, 2020a).
• FAB.
We use the code from (Croce & Hein, 2020a) and keep the hyperparameters the same.
• Square Attack.
We set T = 2000 and the initial fraction of the elements p = 0 .
The other hyperparameters are the same as in (Andriushchenko et al., 2020).
For the adaptive versions, we set the interval u
= (cid:100) T/ 5 (cid:101) .
D. S ELF - SUPERVISED T ASKS
Rotation Prediction is a widely used self-supervision task proposed in (Gidaris et al., 2018) and has been employed in AT as an auxiliary task to improve the robust accuracy (Chen et al., 2020a;
Hendrycks et al., 2019).
Following (Gidaris et al., 2018), we create 4 copies of the input image by rotating it with Ω = { 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ } .
The task then consists of a 4-way classiﬁcation problem, where the head g rotate aims to predict the correct rotation angle.
The loss for an image x is the average
[Equation]
(37) where x ω is the rotated image with angle ω ∈ Ω , G rotate = g rotate ◦ E denotes the classiﬁer for rotation prediction, and G rotate ( · ) ω is the predicted probability for the ω angle.
The head g rotate is a fully-connected layer followed by a softmax layer.
Vertical Flip (VFlip) Prediction is a self-supervised task similar to rotation prediction and has also been used for self-supervised learning (Saito et al., 2020).
In essence, we make two copies of the input image and ﬂip one copy vertically.
The head g vﬂip then contains a 2-way fully-connected layer followed by a softmax layer and predicts whether the image is vertically ﬂipped or not.
The corresponding loss for an image x is
[Equation]
where V = { ﬂipped , not ﬂipped } is the operation set and G vﬂip =
g vﬂip ◦ E .
x v denotes and transformed input and G vﬂip ( · ) v is the probability of operation v .
Note that we do not ﬂip the image horizontally as it is a common data augmentation technique and classiﬁers typically seek to be invariant to horizontal ﬂip.
E A DAPTIVE A TTACKS
In the white-box attacks, the attacker knows every detail of the defense method.
Therefore, we need to assume that the attacker is aware of our test-time adaptation method and will adjust its strategy for generating adversarial examples accordingly.
Here, we discuss one such strong adaptation strategy targeted to our method.
Suppose that the attacker is fully aware of the hyperparameters for test-time adaptation.
Then, ﬁnding adversaries (cid:101) B (cid:63) of the clean subset (cid:101) B can be achieved by maximizing the adaptive loss
[Equation]
(39) where L attack refers to the general attack loss, such as the cross-entropy or the difference of logit ratio (DLR) (Croce & Hein, 2020a).
Let θ T be the ﬁne-tuned test-time parameters using Algorithm 2.
At the k -th step of the attack, it depends on the input (cid:101) B ( k ) =
{ ( (cid:101) x
( k )
[Equation]
(cid:101) y
[Equation]
) } bj =1 via the update
[Equation]
where L test and B are the loss function and subset of training images mentioned in Eqn (6).
As θ T is a function of the input (cid:101) B ( k ) , we can calculate the end-to-end gradient of (cid:101) x
( k ) i ∈ (cid:101) B ( k ) as
∇
[Equation]
x ( k ) i L
[Equation]
( F (
[Equation]
x ( k ) i );
θ T ( (cid:101) B ( k ) )) .
However, θ T goes through T gradient descent steps, and thus calculating the gradient ∇ (cid:101) x ( k )
i θ T ( (cid:101) B ( k ) ) requires T -th order derivatives of the backbone E , which is virtually impossible if T or the dimension of θ E is large.
We therefore approximate the gradient as
[Equation]
which treats θ T as a ﬁxed variable so that high-order derivatives from θ T ( (cid:101) B ( k ) ( (cid:101) x ( k ) i )) can be avoided.
Although this approximation makes Grad( (cid:101) x
( k ) i ) inaccurate, common white-box attacks use projected gradients, which are robust to such inaccuracies.
For example, PGD only uses the sign of the gradient under an (cid:96) ∞ adversarial budget.
Note that solving the maximization in Eqn (12) does not necessarily require calculating the gradient Grad( (cid:101) x ( k ) i ) .
For instance, we will also use Square
Attack (Andriushchenko et al., 2020), a strong score-based black-box attack, to maximize Eqn (12) and generate adversaries for (cid:101) B .
As another approximation to save time, one can also ﬁx θ T for several iterations.
This leverages the intuition that attack strategies often make small changes to the input (cid:101) x , and thus, for the intermediate images in the k -th and ( k + 1 )-th steps, θ T ( (cid:101) B ( k ) ) and θ T ( (cid:101) B ( k +1) ) should be close.
Therefore, a general version of our adaptive attacks only updates θ T every u iterations, with u a hyperparameter.
In Algorithm 4, 6, 5 and 7, we show the algorithms for (cid:96) ∞ norm-based adaptive PGD, AutoPGD,
Square Attack and FAB, respectively.
The main difference between the original and adaptive versions is the target loss function for maximization.
The reader may refer to (Andriushchenko et al., 2020; Croce & Hein, 2020a;b) for a more detailed description of the steps in these algorithms ( e.g. , the condition for decreasing the learning rate in AutoPGD).
for
[Table]
2: θ θ T 3:
(cid:101) x
i
B do 4: (cid:101) x 0 i = (cid:101) x i 5:
Grad( (cid:101) x i ) = ∇ (cid:101) x i L attack ( F ( (cid:101) x i ) , (cid:101) y
i ; θ ) 6: (cid:101) x 1 i =
Clip [ (cid:101) x
i − ε, (cid:101) x
i + ε ] ( (cid:101) x
i + γ Sign(Grad( (cid:101) x i )))
7: l 0 i =
L attack ( F ( (cid:101) x 0 i ) , (cid:101) y
i ; θ ) 8: l 1 i =
L attack ( F ( (cid:101) x 1 i ) , (cid:101) y
i ; θ ) 9: l ∗ i = max { l 0 i , l 1 i } 10: (cid:101) x ∗ i = (cid:101) x 0 i if l ∗ i =
l 0 i else (cid:101) x ∗ i = (cid:101) x 1 i 11: end for 12: for t = 1 to T − 1 do 13: if t mod u = 0 then 14:
Get ﬁnal parameter θ T by taking (cid:101) B ∗ = { (cid:101) x ∗ i } as input image for Algorithm 2.
15: θ = θ T 16: end if 17: for i = 1 , ..., | (cid:101) B | do 18:
Grad( (cid:101) x ti ) = ∇ (cid:101) x ti L attack ( F ( (cid:101) x ti ) , (cid:101) y
i ; θ ) 19: z t +1 i =
Clip [ (cid:101) x
i − ε, (cid:101) x
i + ε ] ( (cid:101) x ti + γ Sign(Grad
( (cid:101) x ti )))
20: (cid:101) x
t +1 i
=
Clip [ (cid:101) x
i − ε, (cid:101) x
i + ε ] ( (cid:101) x ti + ξ ( z t +
i − z ti )
+ (1 − ξ )( (cid:101) x ti − (cid:101) x t − 1 i )) 21: l t +1 i
=
L attack ( F ( (cid:101) x
t +1 i ) , (cid:101)
y
i ; θ ) 22: (cid:101) x ∗ i = (cid:101) x
t +1 i and l ∗ i =
l t +1 i if l t +1 i > l ∗
i 23: if k ∈ W and satisfy the condition of dropping learning rate then 24: γ = γ/ 2 and (cid:101) x t +1 i
= (cid:101) x ∗
i 25: end if 26: end for 27: end for 28: return Adversarial image (cid:101) x
(cid:63)i = (cid:101)
x ∗ i
=
1: Add to and
B
2: for do 3: if t mod then 4:
5: θ
6: end if 7: i in
B do 8: h
←− side schedule
9: δ ←− array of zeros of size w × w × c 10:
Sample uniformly r, s ∈ { 0 , ..., w − h t } ⊂ N 11: for j = 1 , ..., c do 12: ρ ←− Uniform ( − 2 ε, 2 ε ) 13: δ r +1: r + h t ,s +1: s + h t
= ρ · 1 h t × h t 14: end for 15: (cid:101) x new i
=
Clip [ (cid:101) x
i − ε, (cid:101) x
i + ε ] ( (cid:101) x (cid:48)
i + δ ) 16: l new i
=
L attack ( F ( (cid:101) x new i ) , (cid:101) y
i ; θ ) 17: if l new < l ∗ then 18: (cid:101) x (cid:48)
i = (cid:101)
[Table] [Table]
: v = + ∞
[Table]
t do 4:
[Table]
[Table] [Table]
[Table] [Table]
in
[Table]
do
[Table]
=
[Table]
10: δ
[Table]
=
[Table]
: δ
[Table]
: α
[Table]
min (cid:26)
[Table]
+ (cid:107) δ t orig (cid:107) ∞ , α max (cid:27) ∈ [0 , 1] 13: (cid:101) x (cid:48)
i =
proj C (cid:16) (1 − α )( (cid:101) x (cid:48)
i + η δ t )
+ α ( (cid:101) x
i + η δ t orig ) (cid:17) 14: if (cid:101) x (cid:48)
i is not classiﬁed as y i then 15: if (cid:107) (cid:101) x (cid:48)
i − (cid:101) x
i (cid:107) ∞ < v then 16: (cid:101) x
(cid:63)i = (cid:101) x (cid:48)
i 17: v = (cid:107) (cid:101) x (cid:48)
i − (cid:101) x
i (cid:107) ∞ 18: end if 19: (cid:101) x (cid:48) i = (1 − β ) (cid:101) x i + β (cid:101) x (cid:48)
i 20: end if 21: end for 22: end for 23: return Adversarial image (cid:101) x
(cid:63)i