T EST -T IME A DAPTATION AND A DVERSARIAL R OBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies test-time adaptation in the context of adversarial robustness.
We ﬁrst formulate an adversarial threat model for test-time adaptation, where the defender may have a unique advantage as the adversarial game becomes a maximin game, instead of a minimax game as in the classic adversarial robustness threat model.
We then study whether the maximin threat model admits more “good solutions” than the minimax threat model, and is thus strictly weaker .
On the positive side, we show that, if one is allowed to access the training data, then Domain
Adversarial Neural Networks ( DANN ), an algorithm designed for unsupervised domain adaptation, can provide nontrivial robustness in the test-time maximin threat model against strong transfer attacks and adaptive ﬁxed point attacks.
This is somewhat surprising since DANN is not designed speciﬁcally for adversarial robustness (e.g., against norm-based attacks), and provides no robustness in the minimax model.
On the negative side, we show that recent data-oblivious testtime adaptations, in contrast to DANN , can be easily attacked.
We take a step to discuss moving towards adversarially robust test-time adaptation and examine its various implications.
I NTRODUCTION
There is a surge of interest to study test-time adaptation to help generalization to unseen domains (e.g., recent work by Sun et al.
; Wang et al.
Nado et al.
At the high level, a generic test-time adaptation can be modeled as an algorithm Γ which accepts an (optional) labeled training dataset D , an (optional) model F trained on D (usually used as a starting point), and an unlabeled test feature set U , outputs a model (cid:101) F = Γ( F, D, U ) , in order to achieve high test accuracy on U .
For large test set U , test-time adaptation can be viewed as a form of transductive learning (Joachims (1999); Vapnik (1998))
(i.e., using D, U to train a model to predict speciﬁc instances in
U ), which is argued to be easier than more traditional inductive learning.
This paper studies test-time adaptation in the context of adversarial robustness (i.e., there is an active agent who tries to fool the test-time adaptation by perturbing the input so that (cid:101) F gives wrong predictions).
There are several motivations in pursuing this direction.
First, this question is of practical interest: Many practical ML pipelines run in a batch mode 1 , where they ﬁrst collect a set of unlabelled data points, and then send them to a model (e.g. Nado et al.
In such cases, data in the batch may have been adversarially perturbed, and it is a natural question whether we can leverage the large batch size and test-time adaptation to enhance adversarial robustness.
Second, from a purely theoretical point of view, since test-time adaptation is a form of transductive learning, it is intriguing to ask whether transductive adversarial learning can be easier , given that traditional adversarial robustness is formulated in the inductive learning setting (e.g. Madry et al.
To this end, a recent work by Goldwasser et al.
(2020) shows that, with transductive learning, one can achieve nontrivial guarantees for classes of bounded VC dimension with arbitrary train and test distributions.
The current work complements their paper in the setting of deep learning.
To study these questions, we ﬁrst formalize a threat model, which we call (test-time) maximin threat model , for the adversarial robustness of test-time adaptation.
Recall that the classic adversarial robustness game is a minimax game min F E V [max (cid:101) V L ( F, (cid:101) V )] , where V is random sampled data,
(cid:101) V is the perturbed data generated from V by the adversary, and L ( F, (cid:101) V ) is the loss of the model
F on (cid:101) V .
By contrast, in the maximin threat model, we allow V to be sampled from a different domain, and the game is maximin:
E V [max U min (cid:101) F L ( (cid:101) F, (cid:101) V )]
(where U is the perturbed features of
V , subject to the attack type, and (cid:101) V is the labeled perturbed data, see Deﬁnition 2).
By the maximin inequality, it follows that this threat model is no harder than the minimax model (to allow source and target domains to be different, we need to generalize the classic minimax model, see Deﬁnition 3).
We then move on to the focus of this work: Whether the maximin threat model is “strictly weaker” than the minimax threat model.
We note that any good defender solution (a robust model) in the minimax game induces a good defender solution in the maximin game (an adaptation algorithm that outputs that robust model), thus intuitively, the good defender solutions of the minimax model is a subset of the good defender solutions of the maximin threat model.
We ask whether such a containment is proper : That is, whether there exists a defender solution that is good in the maximin threat model (i.e., provides robustness), but is bad in the minimax threat model (provides no robustness).
The existence of such a defender will demonstrate that the maximin threat model admits more good solutions.
Besides theoretical interest, this question is also of practical importance since these “new” solutions may possess desirable properties that good solutions in the minimax threat model may lack .
For example, one such property is that the defender solution is attack agnostic (Goodfellow (2018) (pp.30)): That is, the solution is not to directly optimize the performance measure for a particular type of perturbation 2 .
On the positive side, we consider (cid:96) ∞ attacks (more results on (cid:96) 2 attacks are reported in the appendix), and empirically show that DANN (Ganin et al.
(2017)), an algorithm designed for unsupervised domain adaptation (UDA), is a candidate that exhibits such a separation between the maximin and minimax models.
This is somewhat surprising as DANN is also agnostic to (cid:96) ∞ attacks.
Speciﬁcally, we generate an adapted model (cid:101) F = DANN ( RI , D, U ) to classify U , where RI is random initialization, D is the labeled training set, and U is generated by either transfer attacks using Projected
Gradient Descent ( PGD ) attacks (Madry et al.
(2018)), or adaptive ﬁxed point attacks.
In the homogeneous case where the train and test distributions are the same, on CIFAR10 for example, this achieves about 50% test accuracy, comparable with robustness achieved by an adversarially trained
CIFAR10 model (which is trained speciﬁcally for (cid:96) ∞ attacks) in the minimax model.
In the inhomogeneous case, on CIFAR10 → CIFAR10c-fog for example, DANN provides about 52% accuracy, signiﬁcantly larger than the 17% accuracy using an adversarially trained CIFAR10 model (note that because the defender does not see clean labeled CIFAR10c-fog data, one cannot hope to use an adversarially trained CIFAR10c-fog model).
Not surprisingly, the accuracy becomes very low when we evaluate (cid:101) F in the minimax model, conﬁrming that it is speciﬁcally for the maximin model.
On the negative side, we explore the maximin robustness of the recent data-oblivious adaptation algorithms (namely, the adaptation algorithms do not use D , but just the pretrained model F and unlabeled test set U ).
Speciﬁcally, we consider Test-Time Training ( TTT ) by Sun et al.
We show that TTT can be easily attacked using simple transfer attacks.
While this is not surprising as authors of Sun et al.
(2020) have cautioned that TTT is not designed for adversarial robustness, the situation is in sharp contrast to our results with DANN .
The rest of the paper is organized as follows:
Section 2 presents necessary setup.
In Section 3 we deﬁne the maximin threat model, and adversarial semi-supervised minimax threat model (a generalization of the classic minimax threat model).
In Section 4 we show that DANN provides, somewhat surprisingly, adversarial robustness in the maximin model.
Finally, Section 5 explores the maximin robustness of oblivious test-time adaptation, and concludes the paper with future directions.
P RELIMINARIES
Let F be a model, for a data point (
x, y ) ∈
X × Y , a loss function (cid:96) ( F ; x, y ) give the loss of F on x given the true label y .
Let V be a set of labeled data points.
We use the notation L ( F, V ) =
1 | V | (cid:80) ( x
,y ) ∈ V (cid:96) ( F ; x, y ) to denote the empirical loss of F on V .
For example, if we use binary loss (cid:96) 0 , 1 ( F ; x, y ) = 1 [ F ( x ) (cid:54) =
y ] , this gives the test error of F on V .
We use the notation V | X to denote the projection of V to its features, that is { ( x i , y i ) } mi =1 (cid:55)→ { x 1 , . . .
, x m } .
Threat model for classic adversarial robustness.
To formulate the threat model for test-time adaptation, we ﬁrst present a threat model for the classic adversarial robustness.
Although the classic adversarial robustness can be written down succinctly as a minimax objective, namely min F E ( x,y ) ∼ ( X,Y ) (cid:2) max x (
cid:48) ∈ N ( x ) [ (cid:96) ( F ; x (cid:48) , y )] (cid:3) ( N ( x ) is a neighborhood function of x , determined by the attack type), a threat model formulation will help us develop more nuanced models.
Deﬁnition 1 ( Threat model for
[Table] [Table]
and
[Table] [Table]
[Table]
learning
[Table] [Table]
• A (labeled) training set D is sampled i.i.d.
from from ( X, Y ) .
Training time • ( Defender ) Train a model F on D as F = T ( D ) .
Test time • A (labeled) natural test set V is sampled i.i.d.
from ( X, Y ) .
• ( Attacker ) On input F , D , and V , A perturbs each point ( x
, y ) ∈ V to ( x (cid:48) , y ) (subject to the agreed attack type), giving (cid:101) V
=
A ( F, D, V ) .
[Table]
We stress that the i.i.d sampling of V is important (which is also present in the expectation in the minimax objective): Otherwise an attacker can pick any point that fools F and repeat it arbitrarily many times.
(we refer readers to Goodfellow (2019) for more discussions along this line).
Notations for models and attacks.
In this paper we mainly use the PGD attacks (Projected Gradient
Descent attacks) with norm-based perturbations Madry et al.
For example, given a model
F , we use the notation PGD ( F, V ) to denote PGD attacks against F , on data V (the attack type is speciﬁed in the context).
We adopt the following notations:
[Table]
T A target model trained on the labeled target data V .
AdvT
An adversarially trained target model using the labeled target data V .
S
A source model trained on the labeled source data D .
AdvS
An adversarially trained source model using the labeled source data D .
PGD ( · , · ) PGD Attacks on a model and data.
For example, PGD ( AdvT , V ) means to use PGD attacks on the model AdvT and data V .
Test-time defenses and BPDA.
Various previous work have investigated test-time defenses where a pretrained model is ﬁxed and there is a “preprocessing procedure” to preprocess an input before sending it to the model.
Several such defenses were described and attacked in Athalye et al.
(2018), by the BPDA technique (Backward Pass Differentiable Approximation).
While syntactically one can ﬁt test-time defenses into our framework, the test-time adaptation concept discussed in this paper is signiﬁcantly different, which usually allows one to completely modify a model.
As we will see later in the paper, we use an unsupervised domain adaptation algorithm, such as DANN , for test-time adaptation.
In these cases, it is unclear how to apply BPDA.
T HREAT MODELS 3. T EST - TIME MAXIMIN THREAT MODEL
The intuition behind the test-time maximin threat model is as follows: After we receive the adversarially perturbed data U to classify (at test time), the defender trains a model based on U , and we evaluate the test accuracy only for U .
(i.e., for different test set U we may have different models and different test accuracy.)
This perspective of training a model using labeled data D and unlabeled data U , and only test on U , is essentially the transductive learning by Vapnik (1998) (however, we consider it in an adversarial setting).
Formally, we have the following deﬁnition:
Deﬁnition 2 ( Maximin threat model for test-time adaptation).
Fix an adversarial perturbation type.
We have a source domain ( X s , Y s ) , and a target domain ( X t , Y t ) .
Attacker is an algorithm A , and defender is a pair of algorithms ( T , Γ) , where T is a supervised learning algorithm
and Γ is a test-time adaptation algorithm.
Before game starts
• A (labeled) training set D is sampled i.i.d.
from ( X s , Y s ) .
Training time • (Defender, optional) Train a model F trained on D as F = T ( D ) .
Test time • A (labeled) natural test set V is sampled i.i.d from ( X t , Y t ) , V is sent to the attacker.
• (Attacker) Produce an unlabeled dataset U as follows: 1.
On input Γ , F , D , and V , A perturbs each point ( x
, y ) ∈ V to ( x (cid:48) , y ) (subject to the agreed attack type), giving (cid:101) V
=
A (Γ , F, D, V ) .
Send U =
(cid:101) V | X (that is, the feature vectors of (cid:101) V ) to the defender.
• (Defender) Produce an adapted model as (cid:101) F = Γ( F, D, U ) .
Evaluation: Evaluate the test loss of (cid:101) F on (cid:101) V , L ( (cid:101) F, (cid:101) V ) .
On the adversary: While the adversary knows the adaptation algorithm, Γ may use some private ran- domness, such as random initialization.
Since defender proceeds after the attacker to apply Γ , these private randomness is assumed not to be known by the adversary.
On notations: If there is no need for a pretrained model, we use the notation Γ( ⊥ , · , · ) .
Remarks on the threat model.
We note the following aspects:
Modeling differences.
The ﬁrst obvious distinction is that source domain may differ from the target domain.
This is called a test-time threat model since (cid:101) F is trained based on U =
(cid:101) V | X , and the test error is only evaluated on (cid:101) V (the perturbed test set).
This is in sharp contrast with the classic minimax threat model, where the model F is trained on D , which is independent from V and so (cid:101) V .
This threat model enables us to study whether a large test set (i.e. large | U | ) can beneﬁt a defender for adversarial robustness .
This is in sharp contrast with the classic minimax threat model, where the attacker is granted much power, and can pick points in a one-by-one, or “online”, fashion (i.e., sample a point from the nature, perturb it, and send to the defender).
This is called a maximin threat model since the adversary must move ﬁrst to submit the challenge set U , and the defender then moves based on it.
In fact, this can be written as a maximin game
E V (cid:2) max U min (cid:101) F test err( (cid:101) F, (cid:101) V ) (cid:3) , different from the minimax game in the classic threat model.
Same security goal.
However, in the case where these two domains coincide, the maximin threat model is still nontrivial.
In fact, from the security perspective, both threat models are about the same end-to-end security goal: Correct predictions on adversarially perturbed data.
Attacking a model vs. Attacking an algorithm.
In the classic threat model, a model F is ﬁxed after training, and at the test time the adversary attacks the model.
However, in the maximin threat model, the adversary must attack an adaptation algorithm Γ .
Restrictions of the threat model.
Astute readers may realize that we have intentionally left the deﬁnition of the test-time adaptation Γ to have a lot of freedom.
For example, Γ can leverage the labeled training data D , which may be practically not available or computationally infeasible for test-time computations.
We consider the following possibilities of restrictions:
Homogeneity .
If the source and target domains equal, we call it the homogeneous maximin model .
This setting is directly related to the classic threat model (Deﬁnition 1), where we have a single domain ( X, Y ) , but an adversary can perturb the test instances.
From a security perspective, maximin threat model gives a relaxed threat model, but the same end-to-end security guarantees .
(i.e., correct predictions against adversarially perturbed input)
Data-obliviousness .
In this case, the adaptation algorithm Γ cannot leverage the labeled source data
D .
In other words, the agent can only leverage the pretrained model F and the unlabeled data set U .
3. A DVERSARIAL SEMI - SUPERVISED MINIMAX THREAT MODEL
An unsatisfying aspect of the theory thus far is that the classic minimax threat model is only deﬁned in the homogeneous case, but the maximin threat model can be inhomogeneous, which makes the direct comparison difﬁcult.
To bridge the gap, we provide a generalization of the classic minimax threat model that lies between the classic minimax threat model (Deﬁnition 1) and the test-time maximin threat model (Deﬁnition 2).
Deﬁnition 3 ( Adversarial semi-supervised minimax threat model).
Fix an adversarial perturbation type.
We have a source domain ( X s , Y s ) , and a target domain ( X t , Y t ) .
The attacker is a pair of algorithms ( A 0 , A 1 ) , and the defender is a pair of algorithms ( T , Γ) , where T is a supervised learning algorithm, and Γ is a semi-supervised learning algorithm.
Before game starts
• A training set D is sampled i.i.d.
from ( X s , Y s ) .
• A semi-supervision set V is sampled i.i.d from ( X t , Y t ) , V is sent to the attacker.
Training
time • (Defender, optional) Train a model F trained on D as F = T ( D ) .
• (Attacker) Produce an unlabeled dataset U as follows: 1.
On input Γ , F , D , and V , A 0 perturbs each point ( x, y ) ∈ V to ( x (cid:48) , y ) (subject to the agreed attack type), giving (cid:101) V =
A 0 (Γ , F, D, V ) .
Send U =
(cid:101) V | X (that is, the feature vectors of (cid:101) V ) to the defender.
• (Defender) Produce a ﬁnal model as Γ : (cid:101) F = Γ( F, D, U ) .
Test time • A (labeled) natural test set V (cid:48) is sampled i.i
d. from ( X t , Y t ) .
• (Attacker)
On input (cid:101) F , D , and V (cid:48) , A 1 perturbs each point ( x, y ) ∈ V (cid:48) to ( x (cid:48) , y ) (subject to the agreed attack type), giving (cid:102) V (cid:48)
=
A 1 ( (cid:101) F, D, V (cid:48) ) .
[Table] [Table]
L ( (cid:101) F,
[Table]
V
[Table] [Table]
Remarks on the threat model.
We note the following aspects:
Modeling differences.
(1) This threat model is semi-supervised , because the defender (learner) receives an unlabeled data set U from the target domain.
Note that the procedure to produce (cid:101) F at the training time of this threat model is exactly the same as the procedure to produce (cid:101) F in the test-time maximin threat model.
The key difference is that, once (cid:101) F is trained, we evaluate it on (cid:102) V (cid:48) , which is the adversarially perturbed data on independently sampled V (cid:48) .
This is thus inductive learning, and a minimax game.
This threat model is adversarial , because the attacker can adversarially perturb the clean semi-supervision set V to produce U .
Classic minimax model as a special case.
The classic minimax threat model is a special case of this threat model, by putting source and target domains equal, and choosing a trivial Γ : Γ( F, D, U ) =
F = T ( D ) .
We list several other specializations of this threat model in Appendix B.3.
Therefore, without loss of generality, for the rest of the paper, by minimax model we mean the adversarial semi-supervised minimax threat model .
Γ : One algorithm, two interpretations.
We note that for an algorithm Γ : ( F, D, U ) (cid:55)→ (cid:101) F , one can have now two interpretations: In the maximin threat model, we interpret it as a adaptation algorithm, because we are only going to evaluate (cid:101) F on (cid:101) V ; in the minimax threat model, we interpret it as a semi-supervised learning algorithm, because we are going to evaluate (cid:101) F on unseen points (cid:102) V (cid:48) .
M AXIMIN VS . M INIMAX : A DANN PERSPECTIVE
We now move on to the focus of this work:
Is the maximin threat model “strictly weaker” than the minimax threat model?
Theoretical understanding.
We ﬁrst show that the maximin threat model is no harder than the minimax threat model.
Proposition 1 ( Homogeneous maximin vs. Classic minimax threat model).
Let k ≥ 1 be a natural number, and F be the hypothesis class.
For a given V , the domain of (cid:101) V is a well-deﬁned function of V (e.g., (cid:96) ∞ ball around V ).
We have that:
[Equation]
Note that in the proof (A.1) the homogeneity condition is not essential (one can set ( X, Y ) as the target domain and the proof carries through).
Further, the proof holds verbatim if we extend the family of models in the minimax game to F ( D, U ) .
This thus gives the following corollary,
Corollary 1.
Maximin threat model is no harder than the minimax threat model (Deﬁnition 3).
We also note the following:
Proposition 2 ( Good minimax solution is also a good maximin solution).
Suppose T ∗ is a supervised learning algorithm which trains a model F ∗ =
T ∗ ( D ) , where its adversarial gain in the adversarial semi-supervised minimax model is bounded by κ (i.e. E V
[Equation]
[max
[Equation]
V
[Equation]
L ( F ∗ , (cid:101) V )] ≤ κ. )
Then in the maximin threat model, the adversarial gain of the strategy ( T ∗ , Γ ∗ ) , where Γ ∗ ( F ∗ , D, U ) =
F ∗ =
T ∗ ( D ) , is also upper bounded by κ .
4. A RE GOOD MINIMAX SOLUTIONS STRICTLY CONTAINED IN GOOD MAXIMIN SOLUTIONS ?
Proposition 1 and Proposition 2 lead to the following natural question:
Is there a Γ such that:
• As a test-time adaptation algorithm in the maximin model, it provides robustness, but • As a learning algorithm in the minimax model, the model it produces has no robustness.
The existence of such a Γ provides a separation between the minimax and the maximin threat models, and indicates that the maximin threat model admits more good solutions.
Despite theoretical interest, this question is of practical importance since these “new” solutions may possess desirable properties that good solutions in the minimax threat model may lack .
In this section we consider one such desirable property that the defender solution is attack agnostic (Goodfellow (2018) (pp.30)):
That is, The defender strategy is not to directly optimize the performance measure.
(e.g., we know the attack is (cid:96) ∞ -norm attacks, and we directly train for it).
High-level experiment design .
While we are not able to prove the existence of a defender solution that separates maximin and minimax threat models, we design and implement experiments to provide evidence that such a strategy may actually exist.
The experiment design is as follows:
(1) We use the DANN algorithm with random initialization ( RI ), as a test-time adaptation; that is, DANN ( RI , · , · ) .
There are several motivations of choosing DANN :
(A) DANN ﬁts our framework as it is designed for unsupervised domain adaptation.
(B) DANN is however not designed for adversarial robustness.
Thus is it a very interesting question whether DANN can provide test-time adversarial robustness against attacks (e
g. norm-based attacks) that it is not speciﬁcally designed for .
(C) DANN algorithm leverages source data D , which could beneﬁt the maximin robustness.
(2) We generate adversarially perturbed (cid:101) V , and check whether DANN can provide robustness.
(3) Note that DANN ( RI , · , ·
) can be viewed as a semi-supervised learning algorithm in the adversarial semi-supervised minimax threat model .
Therefore, we check whether the adapted model (cid:101)
F = DANN ( RI , D, U )
is robust in the minimax model.
(4) If (2) and (3) show
that DANN ( RI ,
is
signiﬁcantly more
robust
in the test-time
maximin
threat model than in the minimax model, then the experiment goal is
achieved.
Attacks.
We use (cid:96) ∞ attacks in this section (note that DANN is not designed for (cid:96) ∞ attacks).
We report more results about (cid:96) 2 attacks in Appendix E.3.
We consider two categories of attacks:
Transfer Attacks .
Transfer attacks are a common class of attacks, where we transfer attacks on one model to the target model (in our context, produced by a maximin defender strategy).
In this paper we will mainly apply PGD attacks on adversarially trained models to generate transfer attacks.
Fixed Point Attacks .
We also consider adaptive attacks where an adversary can leverage the knowledge of the adaptation algorithm Γ .
We propose a new family of adaptive attacks called ﬁxed point attacks ( FPA ) .
FPA is developed based on a bilevel optimization formulation of the adversarial game with test-time adaptation based defenses, and adapts previous bilevel optimization algorithms for deep nets, proposed in Lorraine & Duvenaud (2018), to the test-time adaptation setting.
The intuition (we refer readers to Section D for details) behind FPA is that since our goal is to ﬁnd U on which Γ( F, D, U ) fails (thus U is a “ﬁxed point”), we thus start with a U 0 , and iteratively compute
F i =
Γ( F, D, U i − 1 ) , generate U i by attacking F i , and hope that with large number of iterations we can converge to a good U
= U k .
Formally, FPA (Algorithm 1) accepts k, F as parameters, and instantiate U 0 by attacking F , and then repeatedly attacking the models produced by Γ .
Datasets.
For the homogeneous case, we consider MNIST and CIFAR10 (i.e., for both source and target domains).
The homogeneous case represents the security problem considered in the classic threat model.
For the inhomogeneous case, we consider MNIST → MNIST-M (Ganin et al., 2017), and CIFAR10 → CIFAR10c-fog (Hendrycks & Dietterich, 2019).
For CIFAR10c-fog, all 5 levels combined and perform and 80 - 20 train test split, to obtain the training set of size 40000 and test dataset of size 10000 (for space reasons, we combine all corruption levels for experiments in this section, results where we separately study different levels are reported in Section E.2).
Models.
For MNIST, we use the original DANN architecture from Ganin et al.
(2017) with slight modiﬁcations (e.g. adding batch normalization and dropout layers).
For CIFAR10, we use preactivation ResNets from He et al.
(2016) for the prediction branch of DANN , and for the domain prediction branch
we use the architecture from Sun et al.
(2020) (convolution layer).
This architecture is slightly different from the the typical DANN architecture used for MNIST, which assumes vectorized feature and fully connected domain prediction branch.
[Table]
In all cases, DANN demonstrated better accuracy (albeit in a weaker threat model).
Experiment results.
(A) Table 1 summarizes the results under transfer attacks.
In the homogeneous case, the adversarial accuracy DANN provided in the maximin model is comparable to the adversarial accuracy an adversarially trained model provided in the minimax model.
And the adversarial
[Table]
[Figure]
We plot accuracy as we train more epochs with DANN during the adaptation, for different target size.
Adversarial lower accuracy.
accuracy becomes signiﬁcantly higher in the inhomogeneous case (compared to using an adversarially trained source model).
(B) Table 2 summarizes the results under FPA attacks.
Similar to the transfer attack case, DANN provides noticeable robustness.
Note that since the defender moves after the attacker, he or she always applies adaptation “one more time”.
(C) Figure 1 gives the robustness results under different target size.
As we can see, the robustness of DANN degrades as the target size decreases, conﬁrming our intuition that large target size beneﬁts test-time robustness.
(D) Finally, we also evaluate the accuracy of the adapted DANN models in the minimax threat model.
Not surprisingly, the accuracy becomes very low.
This thus concludes our experiment that DANN provides robustness in the test-time maximin threat model, but not in minimax threat model.
T OWARDS ADVERSARIALLY ROBUST TEST - TIME ADAPTATION
We brieﬂy explore the adversarial robustness of the recent data-oblivious test-time adaptation algorithms.
Speciﬁcally, we focus on the TTT algorithm by Sun et al.
Recall that a test-time adaptation algorithm is data-oblivious, if it does not use the labeled training data D at the test time.
TTT algorithm uses a specially trained (with self training) pretrained model, which we denote as
PrTTT .
Similar to our DANN experiment, we conduct experiments on CIFAR10 → CIFAR10c-fog: (1) We found that attacking PrTTT can already break TTT algorithm.
While this is not surprising as authors of Sun et al.
(2020) have cautioned that TTT is not designed for adversarial robustness, this is in sharp contrast to our results with DANN .
(2) We then use an adversarially trained model as the pretrained model for TTT .
We found that this indeed increases the maximin-robustness of TTT .
However, the robustness is roughly the same (about 1% difference) as directly using the pretrained model.
This indicates that the robustness mainly comes from the adversarially trained model, and
TTT algorithm provides no robustness.
This is again in sharp contrast to the situation of DANN .
We list several implications as future directions:
• Is transductive adversarial deep learning easier?
Both the current work and the work of Goldwasser et al.
(2020) have given indication that adversarial robustness may be easier in the transductive learning setting.
Can we provide more theoretical evidence, in particular, for deep learning?
• The surprising power of distribution matching for robustness.
It is somewhat surprising that a distribution matching algorithm, such as DANN , can provide adversarial robustness (albeit in a weaker threat model), even when the target domain is already a corrupted domain such as CIFAR10c-fog.
This means that for practical applications where we run an ML model in a batch mode on some collected unlabeled data, one can consider ﬁrst apply UDA to produce an adapted model and then classify the batch, and this may provide adversarial robustness.
To this end, can we certify such robustness?
Or can we ﬁnd better attacks?
• Does robust test-time adaptation always need to access source data?
We note that while DANN provides maximin robustness, it needs to access the labeled training data D at test time, and while TTT does not need D , it provides no maximin robustness.
Can we achieve the best of both of worlds and get adversarially robust oblivious test-time adaptation algorithms?
A P ROOFS A. P ROOF OF P ROPOSITION
Let F be the family of models (cid:101) F we can choose from.
From the maximin inequality, we have that
[Equation]
Note that for the minimax, the max over (cid:101) V is also constrained to perturb features (as we want to ﬁnd adversarial examples).
If we take expectation over V
[Equation]
we have then
[Equation]
Note that
[Equation]
which completes the proof.
A. P ROOF OF P ROPOSITION
In the maximin threat model, let us take the strategy ( T ∗ , Γ ∗ ) as deﬁned.
Therefore the maximin adversarial gain is
[Equation]
which, by assumption, is bounded by κ .
B M ORE ON THREAT MODELS B. O N THE MODELING
Why is the adversary constrained to attack each point in Deﬁnition 2 and Deﬁnition 3?
This is set up as the rule of the threat model, because if not, an adversary can pick any point that fails the model and repeats it arbitrary number of times, which trivializes the model.
In fact, such one-by-one attack is implicit in the original minimax objective, and is also the common practice of evaluating adversarial robustness (i.e., typical evaluation is to take a test set, attack the points one by one, and then evaluate adversarial accuracy).
Our threat model formulation makes this hidden assumption explicit.
We refer readers to Goodfellow (2019) for more discussion if one wants to break the i.i.d.
sampling assumption.
Why in the minimax modeling, we do not assume A in Deﬁnition 1, or A 1 in Deﬁnition 3 to know the training algorithms?
For example, T is not in the input of A in Deﬁnition 1.
This is the case because in the inner most expression L ( F, (cid:101) V ) only depends on F , and F is ﬁxed after the training.
Therefore, bringing in the training algorithm gives no additional information (note that this is also the typical practice of evaluating adversarial robustness in the classic setting, where we just attack the model).
By contrast, this is not the case for the maximin threat model because attacker moves ﬁrst and the model is not ﬁxed yet, so in that case, a strong white-box attacker may need to take into consideration the adaptation algorithm, not just the pretrained model.
Why do we need to have two algorithms A 0 , A 1 in Deﬁnition 3?
A 0 attacks the semi-supervised learning algorithm, and thus has Γ as input.
On the other hand, A 1 attacks a model, so neither Γ nor
T is in the input.
Therefore, these two algorithms are different and are separated in the deﬁnition.
Note that the joint goal of A 0 , A 1 is to fail the ﬁnal model on (cid:102) V (cid:48) , so A 0 can be totally different from
A 1 in order to fail the training.
In this respect, A 0 is like data poisoning the unlabeled data set in the semi-supervised learning.
Unlike data poisoning though, we have the constraint that it must attack each point in V .
There are two algorithms A 0 , A 1 in Deﬁnition 3, wouldn’t A 0 help the defender because it leaks information about the target distribution.
We have shown that the defender can choose to ignore the unlabeled data set, and this degenerates to the classic threat model in the homogeneous case.
Otherwise, the semi-supervised learning setting may indeed help the defender, because the information about the target domain (or more unlabeled data if we are in the homogeneous case) is leaked to the defender.
This is, however, exactly why in these cases that this threat model lies between the classic threat model and the maximin threat model.
In the maximin threat model, the attacker leaks U to the defender, and the defender is only required to predict U .
In the semisupervised threat model, things becomes more challenging, because ﬁnally we want to evaluate the model on independent samples.
In the experiments, we essentially instantiated the semi-supervised threat model, for example, as ( A 0 = FPA , A 1 = PGD ) .
Is that too weak?
A strong adversary of course may want to use a strong attack A 0 to fail the training, and thus fail the ﬁnal model, and choosing FPA may indeed be a weak choice.
However, we note that our goal in Step (3) of the experiment design is to show that we can break the adapted model, and ( FPA , PGD ) already full-ﬁlls this purpose.
As another note, in the maximin threat model, currently FPA is the most natural adaptive attack we can think of.
Exploring stronger attacks is an intriguing future direction.
B. M ORE EXTENSIONS AND RESTRICTIONS FOR MAXIMIN THREAT MODEL
Singleton .
In the standard model, we allow U to be a batch which consists of many point.
In the singleton restriction, U is restricted to have only one point (for example, the test-time training algorithm by Sun et al.
(2020) works in this setting).
Deﬁnition 4 ( Online version).
In the online maximin model, we assume that we will sequentially receive batches of test samples, one after another, and we allow Γ to leverage the previous samples received.
We assume, for simplicity of this work however, that the batches are sampled independently from each other.
For the online maximin model, data obliviousness means that the agent cannot directly leverage the previous unlabeled samples he or she received.
However, the agent can save the previous model parameters and use that.
B. S PECIALIZATION OF THE ADVERSARIAL SEMI - SUPERVISED MINIMAX THREAT MODEL
Adversarial training considered in Miyato et al.
Shu et al.
The setting becomes a special case of our general minimax threat model by:
• Instantiating A 0 as a trivial adversary that does not do any perturbation.
That is the defender receives a set of clean set U of unlabeled features.
• The defender adaptation strategy does adversarial training on the unlabeled data, given by
[Equation]
where r ∗ =
argmax
[Equation]
Unsupervised Adversarial Training Uesato et al.
This is similar to the above except that the source and target domains are equal.
C R ELATED W ORK
Adversarial robustness.
Adversarial robustness of deep learning has received signiﬁcant attention in recent year.
By far a standard approach is to do adversarial training directly for the attack type Madry et al.
; Sinha et al.
Several recent works have studied adversarial training in a semi-supervised setting Carmon et al.
; Uesato et al.
As we have discussed, these work can be viewed as instantiations in the adversarial semi-supervised minimax threat model.
Our work can be viewed as one step further to study adversarial robustness in the transductive setting for deep learning.
Test-time adaptation and transductive learning.
Transductive learning is a long line of research (for example, see the classic work of Transductive Support Vector Machine Joachims (1999)), where one attempts to leverage unlabeled data U to predict on speciﬁc instances of U Vapnik (1998).
On the other hand, transductive learning has not received much attention in the deep learning setting.
The recent proposals of test-time adaptation Sun et al.
; Wang et al.
Nado et al.
(2020) seems to be an reincarnation of this idea, with the additional twist of leveraging a pretrained model.
Our work considers test-time adaptation in an adversarial deep learning setting, which to the best of our knowledge, is new.
Unsupervised Domain Adaptation.
A classic approach for analyzing domain adaption is based on H -divergence Kifer et al.
; Blitzer et al.
; Ben-David et al. (2010).
That theoretical framework is the basis for a line of methods that uses adversarial training with neural networks to learn representations that are indistinguishable between source and target domain, in particular domain adversarial neural network (DANN)
Ajakan et al.
; Ganin et al.
and
related techniques Pei et al.
; Zhao et al.
Some other approach used different divergence notions, such as MMD Long et al.
(2014; 2015), Wasserstein distance Courty et al.
; Shen et al.
(2018), and R´enyi divergence Mansour et al.
D M ORE ON FIXED POINT ATTACKS
In this section we present details about the derivation of the ﬁxed point attacks, which is a new family of adaptive attacks we designed to break test-time adaptation based defenses.
To start with, in the maximin threat model, for a ﬁxed D (labeled training set) and V (labeled test set), the game is max U min (cid:101) F L ( (cid:101) F, (cid:101) V ) (the adversary can only modify features, namely U , without modifying the labels).
We focus on norm-bounded attacks, even though the test-time defense strategy can be agnostic to the attack type (i.e., the adaptation algorithm does not explicit leverage the attack type information).
More speciﬁcally, the constraint of the adversary is that he can only perturb the feature of each point in V using norm-based perturbation: Namely for each ( x, y ) the adversary can generate ( x (cid:48) , y ) where (cid:107) x − x (cid:48) (cid:107) ≤ ε .
we use the notation N ( V ) to denote the neighborhood of V which includes all such legitimate perturbed datasets.
In our settings, the adaptation algorithm Γ is typically an optimization objective (which revises the model, we will demonstrate an instantiation using DANN below).
We assume that this objective is
L tta ( F, D, U ) , where F is a pretrained model, D is the labeled data set from source domain, and
U = (cid:101) V | X is the test features (of (cid:101) V ) from the target domain.
Under these conditions, the maximin game can then be written as a bilevel optimization as follows:
[Equation]
In words, the outer optimization is the adversarial game where the adversary tries to ﬁnd (cid:101) V to fool the defense, while the inner minimization solves the test-time adaptation objective L tta , leveraging the information of U = (cid:101) V | X , in order to derive a revised model (cid:101) F ∗ for predictions.
Instantiation using DANN .
If the inner test-time defense DANN, then we know that (cid:101) F can be written as f ◦ φ , and we can write the following bilevel optimization objective, where the inner minimization is the DANN objective Ganin et al.
[Equation]
V ∈ N ( V )
L ( (cid:101) F, (cid:101) V ) subject to: (cid:101) F ∈ arg min φ,f (cid:110) L ( f ◦ φ,
[Equation]
D ) ,
[Equation] [Equation]
V | X ) (cid:17)(cid:111) where d is a distance function, which is realized as a domain discriminator (network).
We refer readers to Ganin et al.
(2017) for more details.
We note that this bilevel optimization formulation, and its concrete instantiation using DANN , give evidence that: (1) Defenses using test-time adaptation are signiﬁcantly different from test-time defenses discussed in Athalye et al.
The game is harder for the adversary because he or she needs to solve a harder optimization problem.
To this end, we note that the test-time defenses discussed in Athalye et al.
(2018) do not amount to bilevel optimizations.
This is because that those defenses are merely about sanitizing input x given a ﬁxed pretrained model, which is known to the adversary at the attack time and can thus be differentiated.
On the other hand, the use of DANN as a test-time adaptation algorithm makes bilevel optimization essential to the adversarial game.
Solving the bilevel optimization (1).
An immediate question is how to solve (1).
We note that this is in general a very difﬁcult task for the attacker (which is good news for the defender!), given that we use deep nets in both the outer and inner optimizations.
The ﬁxed point algorithm proposed in Algorithm 1 is based on the work of Lorraine & Duvenaud (2018), which we speciﬁcally adapt their Algorithm 2 (optimization of hypernetwork, then hyperparameters) to our test-time adaptation setting.
More speciﬁcally, a direct adaptation of their algorithm is to run a single iteration of the following two steps: (1) Solve the inner minimization and get F 0 = arg min (cid:101) F L tta ( F, D, U
= V | X ) , where V is the naturally sampled test set (the ﬁrst step at the test time in the maximin threat model).
(2) Attack F 0 on V to get adversarially perturbed (cid:101) V (using the objective L ( F 0 , (cid:101) V ) of the outer maximization).
Our proposed algorithm 1 extends this to multiple iterations.
E M ORE EXPERIMENTS E. B ASELINES : A CCURACY OF ADVERSARIALLY TRAINED MODELS .
[Table]
See Table 3 for the results.
E. T RANSFER ATTACKS ON DIFFERENT CORRUPTION LEVELS
[Table]
See Table 4 for the results.
E. T RANSFER ATTACKS WITH (cid:96) 2 ATTACKS
Homogeneous ( S = T )
[Table]
See Table 5 for the results.
Plots of ﬁxed point attacks See Figure 2 for the results.
[Figure]