A DVERSARIALLY -T RAINED
D EEP
N ETS
T RANSFER
B ETTER
O
N
I
MAGE R
ECOGNITION
Anonymous authors
Paper under double-blind review
A BSTRACT
Transfer learning has emerged as a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains.
This process consists of taking a neural network pre-trained on a large feature-rich source dataset, freezing the early layers that encode essential generic image properties, and then ﬁne-tuning the last few layers in order to capture speciﬁc information related to the target situation.
This approach is particularly useful when only limited or weakly labeled data are available for the new task.
In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models, especially if only limited data are available for the new domain task.
Further, we observe that adversarial training biases the learnt representations to retaining shapes, as opposed to textures, which impacts the transferability of the source models.
Finally, through the lens of inﬂuence functions, we discover that transferred adversarially-trained models contain more human-identiﬁable semantic information, which explains – at least partly – why adversarially-trained models transfer better.
I NTRODUCTION
While deep neural networks (DNNs) achieve state-of-the-art performance in many ﬁelds, they are known to require large quantities of reasonably high-quality labeled data, which can often be expensive to obtain.
As such, transfer learning has emerged as a powerful methodology that can signiﬁcantly ease this burden by enabling the user to adapt a pre-trained DNN to a range of new situations and domains (Bengio, 2012; Yosinski et al., 2014).
Models that are pre-trained on ImageNet (Deng et al., 2009) have excellent transfer learning capabilities after ﬁne-tuning only a few of the last layers (Kornblith et al., 2019) on the target domain.
Early work in transfer learning was motivated by the observation that humans apply previously learned knowledge to solve new problems with ease (Caruana, 1995).
With this motivation, learning aims to extract knowledge from one or more source tasks and apply the knowledge to a target task (Pan & Yang, 2009).
The main beneﬁts include a reduction in the number of required labeled data points in the target domain (Gong et al., 2012; Pan & Yang, 2009) and a reduction in training costs as compared to training a model from scratch.
However, in practice, transfer learning remains an “art” that requires domain expertise to tune the many knobs of the transfer process.
An important consideration, for example, is which concepts or features are transferable from the source domain to the target domain.
The features which are unique to a domain cannot be transferred, and so an important goal of transfer learning is to hunt for features shared across domains.
It has recently been shown that adversarially-trained models (henceforth denoted as robust models) capture more robust features that are more aligned with human perception, compared to the seemingly patternless features (to humans, at least) of standard models (Ilyas et al., 2019).
Unfortunately, these models typically have a lower generalization performance on the source domain, as compared to non-adversarially-trained (i.e., natural) model.
Hence, Ilyas et al.
(2019) hypothesize that nonrobust features that are lost during adversarially training may have a signiﬁcant positive impact on generalization within a given dataset or domain.
This inherently different feature representation between models constructed with adversarial training and models trained with standard methods would also explain why accuracy and robustness are at odds (Tsipras et al., 2019).
This leads to the [Figure] question of whether models that use robust representations generalize better across domains.
This is the main question we address.
In this work, we demonstrate that robust models transfer better to new domains than natural models.
To demonstrate this, we conduct an extensive number of transfer learning experiments across multiple domains (i.e., datasets), with various numbers of ﬁne-tuned convolutional blocks and random subset sizes from the target dataset, where the critical variable is the constraint used to adversarially train the source model.
(Described in detail in Sections 3 and Appendix A.3.)
Importantly, note that we do not use an adversarial training procedure for the actual transfer learning process.
Our ﬁndings indicate that robust models have outstanding transfer learning characteristics across all conﬁgurations, where we measure the performance in terms of model accuracy on target datasets for varying numbers of training images and epochs.
Figure 1 provides a summary of our approach.
Our focus in this work is to show that robust source models learn representations that transfer better to new datasets on image recognition tasks.
While adversarial training was proposed to combat adversarial attacks, our experiments discover an unintended but useful application.
Adversarial training retains the robust features that are independent of the idiosyncrasies present in the source training data.
Thus, these models exhibit worse generalization performance on the source domain, but better performance when transferred.
This observation is novel, and we undertake extensive empirical studies to make the following contributions:
• We discover that adversarially-trained source models obtain higher test accuracy than natural source models after ﬁne-tuning with fewer training examples on the target datasets and over fewer training epochs.
• We notice that the similarity between the source and target datasets affects the optimal number of ﬁne-tuned blocks and the robustness constraint.
• We show that adversarial training biases the learned representations to retain shapes instead of textures, impacting the source models’ transferability.
• We interpret robust representations using inﬂuence functions and observe that adversarially-trained source models better capture class-level semantic properties of the images, consistent with human concept learning and understanding.
R ELATED W ORKS
ImageNet transfers.
Our focus is on studying the transfer of all but the last few layers of trained
DNNs and ﬁne-tuning the last non-transferred layers.
For ease of exposition, we restrict our attention to ImageNet models (Deng et al., 2009).
Kornblith et al.
(2019) study the transfer of natural models to various datasets, and is thus a prequel to our work.
Yosinski et al.
(2014) also study transferring natural models but focus on the importance of individual neurons on transfer learning.
Recht et al.
(2019) study the generalization of natural and robust models to additional data generated using a process similar to that of generating ImageNet.
They conclude that models trained on ImageNet overﬁt the data.
However, they study the models’ generalization as-is without ﬁne-tuning.
Covariate shift.
A signiﬁcant challenge in transfer learning is handling the data distribution change across different domains, also called covariate shift.
It’s widely recognized in successful domain adaptations (Yosinski et al., 2014; Glorot et al., 2011) that the representations in earlier layers are more “generic” and hence more transferable than the ones in later layers.
This hierarchical disentanglement is attributed to the properties of the data itself, so that the later layers are more closely associated with the data and do not transfer as well.
This motivated studies for shallow transfer learning (Yosinski et al., 2014; Ghifary et al., 2014) and more general studies to extract features that remain invariant across different data distributions (Arjovsky et al., 2019).
In Section 5 we see that adversarial training biases the learned representations to retain shapes instead of textures, which may be a more desirable invariant across the datasets.
Transfering adversarially-trained models.
There are mainly two works directly associated with ours.
First, subsequent to this paper’s initial posting (in a non-anonymized form in a public forum),
Salman et al.
(2020) posted a related paper.
They arrived at broadly similar conclusions, conﬁrming our main results that robust models transfer better; and they do so by focusing on somewhat different experiments, e.g., they focus on the effects of network architecture width, ﬁxed feature transfer, and seeing if models without texture bias transfer better than robust models.
Second, Shafahi et al.
(2020) mainly ﬁnd that models lose robustness as more layers are ﬁne-tuned.
It might seem to contradict our thesis that they also notice that an ImageNet robust model with a (cid:107) δ (cid:107) ∞ ≤constraint has lower accuracy on the target datasets, CIFAR-10 and CIFAR-100, compared to a natural ImageNet model.
However, we show that the robust model transfers better than the natural one when we use a (cid:107) δ (cid:107) 2 ≤ 3 constraint to adversarially train the source model.
Example based interpretability.
There has been signiﬁcant interest in interpreting blackbox models using salient examples from the data.
A line of research focuses on using inﬂuence functions (Koh & Liang, 2017;
Koh et al., 2019; Khanna et al., 2019) to choose the most indicative data points for a given prediction.
In particular, Khanna et al.
(2019) discuss the connection of inﬂuence functions with Fisher kernels; and Kim et al.
(2016) propose using criticisms in addition to representative examples.
Complimentary lines of research focus on interpretability based on human-understandable concepts (Bau et al., 2017) and feature saliency metrics (M. Ancona, 2017).
B RIEF O VERVIEW OF THE A DVERSARIAL T RAINING P ROCESS
Adversarial training modiﬁes the objective of minimizing the average loss across all data points by ﬁrst maximizing the loss produced by each image with a perturbation (i.e., a mask) that may not exceed a speciﬁed magnitude.
Here, we describe this process, similar to Madry et al.
Let ( x i , y i ) be m data points for i ∈ [ m ] , where x i ∈ R
d is the i th feature vector, and
y i ∈ Y is the corresponding response value.
Typically, we model the response as a parametric model h θ : R d → Y with a corresponding loss function (cid:96) :
Y × Y → R ≥ 0 .
The objective is to minimize the loss (cid:96) (ˆ y, y ) , where ˆ y
=
h θ ( x ) is the predicted response.
Adversarial training replaces the above minimization problem of training the model by a minimax optimization problem to make the model resilient to arbitrary perturbations of inputs.
The goal of adversarial training is to solve a problem of the form
[Equation]
That is, the goal is to ﬁnd the parameters θ of the model h
θ that minimize the average maximum loss obtained by perturbing every input x i with a δ i constrained such that its (cid:96) p norm does not exceed some non-negative (cid:15) .
If (cid:15) = 0 , then δ i = 0 , in which case there is no perturbation to the input, which is what we call natural training .
As (cid:15) increases, the magnitude of the perturbation also increases.
For more details on how we solve this problem, and a few examples, see Appendix A.2.
T RANSFERRING A DVERSARIALLY -T RAINED M ODELS
In this study, we train four ResNet50 source models on ImageNet.
We train one of them naturally (non-adversarially), and train each of the remaining three adversarially with one of the following constraints: (i) (cid:107) δ (cid:107) 2 ≤ 3 , (ii) (cid:107) δ (cid:107) ∞ ≤ 4255 , (iii) (cid:107) δ (cid:107) ∞ ≤ 8255 .
Next, we ﬁne-tune some convolutional blocks in the source models to each of the six target datasets separately using a subset of the training data.
We repeat each of these trials for various seed values and report the mean and 95% conﬁdence interval.
Altogether, we have a comprehensive and replicable experimental setup that considers four ImageNet source models, four ﬁne-tuning conﬁgurations, six target datasets, ten random subset sizes, and an average of ﬁfteen random seeds for a total of 14,400 ﬁne-tuned models.
For more details, see Appendix A.3 and A.4.
Adversarially-trained models transfer better and faster.
For ease of comparison, we select the robust and natural models that transfer with the highest test accuracy across all datasets (ﬁne-tuning three convolutional blocks and the robust model using the (cid:107) δ (cid:107) 2 ≤ 3 constraint), as shown in Figures 2 and 3.
See Appendix A.5 for additional results.
Figure 2(b) shows that the test accuracy delta between robust and natural models is above zero for all six target datasets.
Thus, robust models obtain higher test accuracy on the target dataset than the natural model, especially with less training
[Figure] [Figure]
data in the target domain.
Robust models also learn faster, as shown by the positive test accuracy delta in Figure 3(b) for all target datasets after only 11 and 21 ﬁne-tuning epochs.
See Appendix A. for additional information on different random subset sizes.
Fine-tuning cost is the same for both robust and natural models, but training the source model is considerably more expensive. For more detail on computational complexity see A.8. Best results achieved with (cid:96) 2 constraint and ﬁne-tuning one to three convolutional blocks.
Robust models achieve the highest test accuracy on the target datasets when an optimal number of convolutional blocks are ﬁne-tuned, and when these models are trained with an appropriate constraint type.
In particular, ﬁne-tuning zero (only the fully-connected layer) or nine convolutional blocks leads to lower test accuracy than ﬁne-tuning one or three blocks, as shown in Figure 4(a) for all six target datasets.
The natural model and the other two robust models exhibit the same behavior, as shown in Appendix A.7.
To analyze the best constraint type, we select the ﬁne-tuning conﬁguration that yields the highest test accuracy on the target datasets (ﬁne-tuning three convolutional blocks).
We see that the (cid:96) 2 constraint outperforms the (cid:96) ∞ constraint, as shown by the positive accuracy delta between the (cid:96) 2 and (cid:96) ∞ models in Figures 5(d) and (e), respectively. [Figure] [Figure]
Similarity effect on transfer learning conﬁgurations.
Besides noticing that robust models achieved better performance on the target dataset than natural models, we also observe trends in how well they transfer to different datasets.
When transferring from ImageNet, we ﬁnd that CIFARand CIFAR-100 have interesting transfer properties, compared to the other datasets.
In particular, even though all other datasets transfer better when ﬁne-tuning one or three blocks, it seems that models transfer better to CIFAR-10 and CIFAR-100 when fewer blocks are ﬁne-tuned, as shown in
Figure 4(b).
This suggests that because these datasets are close to ImageNet, ﬁne-tuning of early blocks is unnecessary (Yosinski et al., 2014).
Along similar lines, it is better to use a smaller (cid:15) for CIFAR-10 and CIFAR-100 datasets than the other datasets when transferring from ImageNet, as seen from Figure 5(c).
This is because a larger perturbation would destroy low-level features, learned from ImageNet, which are useful to discriminate between labels in CIFAR-10 and CIFAR100.
Finally, for datasets that are most distinct from ImageNet (SVHN and KMNIST), we ﬁnd that robustness yields the largest beneﬁt to classiﬁcation accuracy and learning speed, as seen in Figure 2(b) and Figure 3(b), respectively.
These discrepancies are even more noticeable when smaller fractions of the target dataset are used.
B IAS TOWARDS RECOGNIZING SHAPES AS OPPOSED TO TEXTURES
In this section, we explore the effect of texture and shape bias, as described by Geirhos et al.
(2019), on the robust models’ transferability.
As pointed out by Geirhos et al.
(2019), natural models are more biased towards recognizing textures than shapes.
This is in stark contrast to the human bias of recognizing shapes over textures (Landau et al., 1988).
However, Engstrom et al.
(2019) showed that robust models encode humanly-aligned representations, and we observe (e.g see Figure 1(b)) that these representations persist even after ﬁne-tuning on CIFAR-10.
Adversarially-trained models are less sensitive to texture variations.
Table 1a shows that the robust model outperforms the natural one when only tested on Stylized Imagenet (SIN) and also after ﬁne-tuning only the last fully-connected layer to SIN.
Both models are ResNet50s pre-trained on ImageNet (IN), and the robust model uses a (cid:107) δ (cid:107) 2 ≤ 3 constraint.
Models trained on standard and stylized ImageNet are less sensitive to adversarial attacks.
Table 1b shows that the ResNet50 model trained on both IN and SIN (IN+SIN) outperforms the models trained on just IN on a PGD(3) adversarial test accuracy on IN for various (cid:15) levels.
Models trained on both ImageNet and Stylized ImageNet are more robust to adversarial attacks
[Table]
[Table]
IN
[Table] [Table]
or a low pass ﬁltered (Low-Pass) version of Caltech101.
We lowered the resolution to 32x32 and zeroed out the top 1,024 ( ∼ 14% ) frequencies, respectively.
[Table]
Adversarially-trained models are biased towards low resolution and low frequencies.
We observe that the transferability of robust models is also affected by two input perturbations that destroy, or at least damage, textures. Namely, lowering the resolution of images and applying low pass ﬁlters. To demonstrate this, we use the Caltech101 dataset (Li Fei-Fei et al., 2004). This dataset has 101 labels with 30 high-resolution (224x224 pixels or more) images per label.
The results in Table 2 support our conjecture that robust models use shapes more than textures for classiﬁcation by showing that the robust model obtains a higher test accuracy, in both the low-resolution and low-pass versions of Caltech101, than the natural one.
I NTERPRETING R EPRESENTATIONS USING I NFLUENCE F UNCTIONS
In this section, we use inﬂuence functions (Koh & Liang, 2017) to show that robust representations hold semantic information, i.e., robust DNNs classify images like a human would, through similar-looking examples.
Engstrom et al.
(2019) observed that moving the image in carefully chosen directions in the latent space allows for high-level human-understandable feature manipulation in the pixel space.
They suggest that the bias introduced by adversarial training can be viewed as a human prior on the representations, so that these representations are extractors of high-level human-interpretable features.
It has long been established that humans learn new concepts through concept-representative or similar-looking examples (Cohen et al., 1996; Newell, 1972).
Our focus in the present work is to study whether these representations aid the neural network to learn new concepts (namely image labels) akin to how humans learn concepts.
To study this, we use inﬂuence functions as described by Koh & Liang (2017) (see Appendix A.9 for an overview).
For each test image in the CIFAR-10 dataset, inﬂuence functions allow us to answer the following: What is the inﬂuence of each training image on the model prediction for a given test image?
We ask this question for both the robust and natural models, and compare the results.
In our experiments, we ﬁne-tune the last three blocks with the same 3,200 randomly selected training images.
Also, the robust model uses (cid:107) δ (cid:107) 2 ≤ 3 as the constraint.
Adversarially-trained models have more similar-looking inﬂuential images.
Figure 6 shows that the robust models’ most inﬂuential image is often more perceptibly similar to the test image than the natural models’ most inﬂuential image.
Consider, for example, the test image of the blue car (on the second column).
The robust models’ corresponding top inﬂuential training image is a similar-looking blue car, while the natural model has a red truck.
As a second example, the robust models’ top inﬂuential training image for the orange truck (on the far right) is a similar-looking orange truck, while the natural model has a blue and white truck.
Inﬂuential image labels match test image labels more often in adversarially-trained models.
To quantify the visual similarity described above, we show the inﬂuence values (standardized by their matrix norm) of each training image on each test image, sorted by their label as in Figure 6, for both the natural (left) and robust (right) models in Figure 7(a).
Darker and better-deﬁned blocks [Figure] [Figure] across the diagonal signal that the inﬂuence values are more consistent with the test image label index in the has the
Figure 7(b) further accentuates the difference between the robust model and the natural model.
It displays the percentage of times that the label of the top- k inﬂuential image in the training set matches the label in the test image evaluated.
To better understand this ﬁgure, consider the leftmost point in Figure 7(b) for both models.
This point represents the proportion of the training images corresponding to the darkest dots in each horizontal line (i.e., top-1 inﬂuential training image) in (a) that match the label of the given test image, for robust and natural models separately.
6% of the robust model’s top-1 inﬂuential images match the label of the given test image vs 55 .
1% for the natural counterpart.
We also consider the case when the category of at least three of the topinﬂuential training images match that of the test image.
This happens in 77 .
3% of the cases for the robust model, but only for 53 .
8% of the cases for the natural model.
This vast gap is not explainable solely from only ∼ 5% difference in target test accuracy, shown in Table 7 in Appendix A.5.
From the qualitative and quantitative analysis, we see that the robust model has learned representations with more human-identiﬁable semantic information than the natural model, while the latter relies on less interpretable representations.
In other words, the robust neural network has learned the image labels by creating strong associations to semantically-similar examples (akin to examplebased concept learning in human beings) in its internal representations.
Thus, reinforcing the human prior bias hypothesis in robust representations observed by Engstrom et al.
D O OTHER ADVERSARIAL ATTACKS IMPROVE TRANSFERRABILITY ?
Prior works show that there is a connection between the sensitivity of a neural network to Gaussian noise and its robustness to adversarial perturbations (Weng et al., 2018;
Gilmer et al., 2019).
It has also been suggested that Gaussian perturbations can improve or even replace adversarial training (Kannan et al., 2018).
Further, it has been shown that often only a few PGD iterations are sufﬁcient to obtain a robust model (Madry et al., 2018; Shafahi et al., 2019; Wong et al., 2020).
To better understand these trade-offs, in this section we further explore the transferrability of models trained on ImageNet with random Gaussian noise and one-step of PGD (i.e., PGD(1)) using the same methodology as described in Section 4. For all models, including the Gaussian one, we contraint the perturbation, namely δ , to be (cid:107) δ (cid:107) 2 ≤ 3 in order make a fair comparison across models. See A. for more details on our experimental setup. In the following we discuss our experimental results, summarized in Figure 8, which shows the test accuracy delta of each of the three adversarially-trained models versus the natural one.
Training with more steps is marginally better.
Figure 8 (a) and (b) show that more PGD iterations (i.e., PGD(20) vs PGD(1)) slightly improve transferability.
This is evidenced by the slightly higher
Number of training images
T e s t
A cc u r a c y
D e l t a %
Number of training images
Number of training images
CIFAR-100 CIFARSVHN FMNIST
KMNIST MNIST
[Figure]
[Figure]
test accuracy delta in (a), relative to (b) across all target datasets.
Our emperical result agrees with previous works that are showing that more attacker steps typically only improve adversarial robustness slightly (Madry et al., 2018; Shafahi et al., 2019; Wong et al., 2020).
A targeted adversary is better than a random one.
Comparing Figure 8 (a) and (b) to (c) shows that a targeted adversarial attacks (i.e. PGD vs Gaussian) signiﬁcantly improves transferability relative to a random perturbation.
This is evidenced by the slightly higher test accuracy delta in (a), and (b) relative to (c) across all target datasets.
A random adversary is better than no adversary.
Figure 8 (c) shows us that a random adversarial attack can improve transferability.
This is evidenced by the signiﬁcantly positive accuracy delta in (c) across all target datasets.
Our results agree with prior works showing that training a model by perturbing inputs with Gaussian noise can improve adversarial robustness (Kannan et al., 2018).
C ONCLUSION AND F UTURE W ORKS
We show that robust models transfer very well to new domains, even outperforming natural models.
This may be surprising since robust models generalize worse than natural models within the source domain, and since they were originally designed to protect against adversarial attacks.
We show that robust DNNs can be transferred both faster and with higher accuracy, while also requiring fewer images to achieve suitable performance on the target domain.
We observe that adversarial training biases the learned representations to retain shapes instead of textures, which impacts the source models’ transferability.
We also show that the improved classiﬁcation accuracy is due to the fact that robust models have an implicit bias that enables them to comprehend human-aligned features.
Given the widespread use of DNNs, there is great potential for robust networks to be applied to a variety of high-tech areas such as facial recognition, self-driving cars, and healthcare, but understanding the issues we have addressed is crucial to deliver upon that potential.
We see three interesting directions to pursue in future works: 1) Exploring adversarial constraints, beyond the widely-used (cid:96) 2 and (cid:96) ∞ norms, that may bias the source model to learn more transferable representations For example, we could estimate adversarial perturbations using inﬂuence functions or frequency domain information instead of only pixel domain information.
Considering source datasets, such as Tencent’s Large Scale Multi-Label Image Dataset (Wu et al., 2019), that overcome ImageNet’s shortcomings (Tsipras et al., 2020).
Investigating the effect of similarity within target dataset labels on robust transfer learning.
For more details, see Appendix A.12.
A A PPENDIX A. D ETAILS ON F IGURE 1( A )
We generated Figure 1(a) by maximizing the activations in the penultimate layer on out of distribution images (i.e., images that are are not part of the training and test data) as our starting seeds, following the “feature visualization” technique as implemented in Engstrom et al.
The natural and robust models are pre-trained ResNet-50 models on ImageNet after ﬁne-tuning only the last fully-connected layer on CIFAR-10.
The robust model used the (cid:107) δ (cid:107) 2 ≤ 3 constraint.
A. M ORE DETAILS ON A DVERSARIAL T RAINING
In practice, we solve equation 1 using stochastic gradient descent (SGD) over θ .
More concretely, we include a random sample of training examples in a set B , specify a non-negative learning rate α , calculate the gradient with respect to the parameters of the model ∇ θ , and update θ as follows:
[Equation]
[Equation]
This training process has a sequential nature: it ﬁrst ﬁnds the worst possible perturbation δ ∗ i for each training example ( x i , y i ) in B before updating the parameters of the model θ , where
[Equation]
Problem equation 3 is typically solved using projected gradient descent with k update steps, which is what we call PGD( k ).
In this work, we use PGD(20), which means that we take 20 update steps to solve (3).
Each step iteratively updates δ i by projecting the update onto the (cid:96) p ball of interest:
[Equation]
As an example, consider the case of the (cid:96) 2 norm and let f ( δ i ) =
(cid:96) ( h θ ( x i + δ i ) , y i ) .
If we want to meet the restriction that (cid:107) δ i (cid:107) 2 ≤ (cid:15) , we can pick an update value for δ i whose (cid:96) 2 norm will be at most the learning rate.
This yields the problem:
[Equation]
And in the case of the (cid:96) ∞ norm we have
[Equation]
Thus, we set the learning rate to be equal to α = c · (cid:15) num.
of steps for 1 . 5 < c < 4 in order to ensure that we reach the boundary condition for δ i .
Also we must clip δ i according to the (cid:96) p norm in case that it exceeds the boundary condition.
A. T RANSFER L EARNING E XPERIMENTAL S ETUP D ETAILS
Source models.
For all of our experiments, we use four residual networks (ResNet-50) (He et al., 2016) pre-trained on the ImageNet dataset (Deng et al., 2009), one was naturally-trained (without an adversarial constraint), and the others use PGD(20) and the following adversarial constraints: (1) (cid:107) δ (cid:107) 2 ≤ 3 , (2) (cid:107) δ (cid:107) ∞ ≤ 4255 , (3) (cid:107) δ (cid:107) ∞ ≤ 8255 , where δ is a matrix that contains represents the perturbation applied to the input image as described in Section 3, equation (1).
In addition to the natural
ResNet-50, considered as baseline, we also use three robust networks with various constraints.
For speed, transparency and reproducibility, we do not re-train the source models ourselves.
Fine-tuning procedure.
To transfer our models we copy the entire source model to the target model, freeze the all but the last k convolutional blocks, re-initialize the last fully-connected (FC) layer for the appropriate number of labels, and only ﬁne-tune (re-train) the last FC layer plus 0, 1, 3, or convolutional blocks.
Freezing layers in the neural networks entails permitting forward propagation, but disabling the back-propagation of gradients used during SGD training.
We have four different ﬁne-tuning conﬁgurations, one for each number of ﬁne-tuned convolutional blocks.
Note that the
ResNet model that we consider has residual blocks that are composed of three convolutional layers, i.e., we ﬁne-tune 27 layers plus the fully connected layer when the number of ﬁne-tuned blocks is equal to 9.
(See Section A.4 for a visualization of our ﬁne-tuning process).
Random subsets.
One of the most interesting parts of our experimental setup is that we also explore the test accuracy of our ﬁne-tuned models using randomly chosen subsets of 100, 200, 400, ... , and 25,600 images from the target dataset.
These subsets are constructed using random sampling without replacement, with a minor constraint: all labels must have at least one training image.
For each run of model training, we ﬁx the training data to be a randomized subset of the entire training data.
As the number of images in a random subset decreases, the variance in the validation accuracy of the transferred models increases.
Thus, we repeat the ﬁne-tuning procedure using 20 seeds for every subset with at most 1,600 images, and using 10 seeds for all larger subsets.
We reduce the number of seeds for larger subsets because the inherently lower variance in the validation accuracy doesn’t justify paying the computational cost associated to ﬁne-tuning more seeds.
Target datasets.
We transfer our models to a broad set of target datasets, including (1) CIFAR-100, (2) CIFAR-10, (3) SVHN, (4) Fashion MNIST (Xiao et al., 2017), (5) KMNIST and (6) MNIST.
Since all of these datasets have images at a lower resolution than ImageNet, we up-scale our images with bi-linear interpolation.
In addition, we use common data transform techniques such as random cropping and rotation that are well-known to produce high-quality results with certain datasets.
A. F INE
[Table]
Figure 9 illustrates all four ﬁne-tuning conﬁgurations in our experiments.
Notice how in Subﬁgure 9(d) we unfreeze more than half of the ResNet-50 architecture, thereby testing what occurs as we ﬁne-tune a lot of blocks.
All source models are ﬁne-tuned to all datasets using stochastic gradient descent with momentum using the hyperparameters described in Table 3.
[Table]
The learning rate decays to a tenth of it’s current value every 33 or 50 epochs, which corresponds to 1/3 of the total ﬁne-tuning epochs, as shown in Table 4.
Also, the test accuracy frequency refers to how often is the test accuracy computed, in epochs.
So, for example, if the test accuracy frequency is 20, then we check the test accuracy after epoch 1, 21, 41, ..., 81, and 100.
With regards to the random seeds, we have the following formula to deﬁne the set of seeds used, S k , as a function of the total number of random seeds used, k :
[Equation]
Thus, when we use 20 seeds, as it is the case for the subset of 100 images, we use seeds 20000000, 20100000, ..., 21900000.
Large numbers were used to avoid numerical instability issues that arise with small numbers where their binary representation has too many zeroes.
See Table 5 for additional detail with regards to the source models.
Notice that although adversarially-trained models do worse on the source dataset, they outperform naturally-trained models on the target datasets, as shown in Table 7.
[Table] [Table]
See Table 6 for a high-level overview of all datasets used.
This should serve as a reminder that our source dataset is ImageNet
[Table]
with an extensive 1.2 million training images and 1,000 labels, it serves as a great starting point in our experiments.
All other target datasets have a considerably lower number of training and test images.
[Table]
A. A DDITIONAL R ESULTS
Table 7 reports the test accuracy of all of our source models after ﬁne-tuning three blocks using different numbers of training images on each of the six target datasets.
The average test accuracy is reported for all cases where the model is ﬁne-tuned with less than the entire training set.
The bolded numbers represent the highest test accuracy among source models.
From this table, we can see that the robust models consistently outperform the natural models.
A. A DDITIONAL D ETAIL ON L EARNING F ASTER
The following subsection contains the additional charts that were omitted in Figure 3 in Section 4.
Figure 11 shows the same behavior is observed in all three ﬁgures: It’s sub-optimal to ﬁne-tune
[Table]
[Figure]
either 0 or nine convolutional blocks, as opposed to one or three.
Consistent with our methodology for Figure
[Figure]
we adversarially-train models on ImageNet and then ﬁne-tune various numbers of convolutional blocks using a random sample of images in the target dataset.
A. A DDITIONAL D ETAIL ON THE E FFECT OF THE N UMBER OF F INE -T UNED B LOCKS
The following subsection contains the additional charts that were omitted in Figure 4 in Section 4.
Figure 11 shows the same behavior is observed in all three ﬁgures: It’s sub-optimal to ﬁne-tune
[Figure]
either 0 or nine convolutional blocks, as opposed to one or three.
Consistent with our methodology for Figure 4, we adversarially-train models on ImageNet and then ﬁne-tune various numbers of convolutional blocks using a random sample of images in the target dataset. A. C OMPUTATIONAL C OST
In general, a PGD( k ) adversarial training process as described by Madry, is k orders of magnitude more expensive than natural training.
This can be seen directly from the fact that there are k more iterations in the inner maximization loop of the risk minimization procedure.
However, since only the source model must be trained adversarially, and these source models can be downloaded from publicly available repositories, the marginal computational cost of ﬁne-tuning to a target dataset is perhaps more important.
Fortunately, the computational cost of ﬁne-tuning both robust or natural models is the same.
A. A DDITIONAL D ETAILS ON I NFLUENCE F UNCTIONS
Suppose that (cid:96) : R m × R d → R is a smooth loss function and x 1 , . . .
, x n ∈ R m are our given data.
The empirical risk minimization (ERM) problem takes the form
[Equation]
[Figure] Say θ (cid:63) is the argmin solution of the above optimization problem.
Let’s now consider upweighing a data point x train with (cid:15) ∈ R .
This modiﬁes the learning problem as:
[Equation]
Let θ (cid:63)(cid:15) be the solution of the upweighted problem above.
The inﬂuence of a training data point x train on a test data point x test approximates the change in the function value (cid:96) ( x test , θ (cid:63) )
→ (cid:96) ( x test , θ (cid:63)(cid:15) ) with respect to an inﬁnitesimally small (cid:15) , i.e., when (cid:15) → 0 when x train is upweighed by (cid:15) .
This can be calculated in closed form Koh & Liang (2017) as:
[Equation]
[Equation] where g 1 = ∇ (cid:96) ( x train , θ (cid:63) ) (cid:62) , g 2 = ∇ (cid:96) ( x test , θ (cid:63) ) and H is the Hessian of the loss function ∇ 2 f ( θ (cid:63) ) .
In particular, we ﬁrst compute the Hessian of the source model ﬁne-tuned with 3,200 CIFARimages as the sum of the Hessians of the loss of batches of ﬁve images.
Note that we only use the 3,200 images that were used in the ﬁne-tuning process, since it accurately reﬂects the Hessian of the model.
Then we get H − 1 using the (Moore-Penrose) pseudo-inverse, using its singular-value decomposition and including all singular values larger than 1e-20.
Koh et.
al.
Koh & Liang (2017) discuss optimization speedup techniques to determine the most inﬂuential x train for a given x test at scale.
However, ﬁnding top-
k inﬂuential images is a combinatorial problem for k > 1 .
So, typically a greedy selection of the next top inﬂuential image is made iteratively k times.
Further, selecting multiple images also requires consideration of interaction and group effects.
As such, the top- 5 inﬂuential images are likely to be less representative of actual inﬂuence being asserted than one would expect.
Fisher kernels and inﬂuence functions Khanna et.
al.
Khanna et al.
(2019) recently discovered an interesting relationship between Fisher Kernels and Inﬂuence functions: if the loss function (cid:96) ( · ) can be written as a negative log-likelihood, then at the optimum w (cid:63) , the Fisher dot product between two points is exactly the same as the inﬂuence of those points on each other (note that the inﬂuence is a symmetric function).
In other words, ﬁnding the most inﬂuential data point to a given data point is equivalent to ﬁnding the nearest neighbor of the point in the space induced by the Fisher kernel.
As observed in Section 6 for robust training, most inﬂuential points for a data point tend to be largely the ones belonging to the same label.
This implies that the in the Fisher space, the points with the same label tend to be grouped together.
A. A DDITIONAL DETAILS FOR ADVERSARIAL ATTACKS COMPARISON SECTION
Both the PGD(1) and Gaussian models are ResNet-50’s trained on ImageNet-1K using stochastic gradient descent with momentum and the following hyperparameters: 0.1 learning rate, 128 batch size, 0.9 momentum, 10x learning rate decay, and an equally-spaced (i.e. linear) learning rate decay schedule.
The test accuracy of each of these models is 60.29% and 74.02% for the PGD(1) and
Gaussian model, respectively.
Both models use the (cid:107) δ (cid:107) 2 ≤ 3 adversarial constraint. The PGD(1) model uses one attacker step, with a step size of 6 and the perturbation is initialized at zero. The Gaussian model adds a perturbation for each pixel drawn from a standard Normal distribution N ( µ = 0 , σ 2 = 1) . A. C ODEBASE O VERVIEW
The starting point requires downloading the source ImageNet models, and installing the appropriate libraries.
Next, the user can decide how to ﬁne-tune the source models: individually or in batches.
The train.py ﬁle will allow individual training, while the tools/batch.py ﬁle allows training in batches.
The train.py ﬁle contains 9 parameters that are explained by running the following command: python train.py --help .
Also, the helpers.py and delete big files.py ﬁles under the tools folder contain the logic that supports the train.py ﬁle.
This includes the random subset generator, the ﬁne-tuning procedure, and the data transforms.
Separately, note that when running the batch.py ﬁle, the ﬁne-tuned models won’t be saved into the results/logs directory.
This is due to the fact that models can occupy a signiﬁcant amount of memory and we do not plan to use these ﬁne-tuned models in the future.
However, if the user wants to save the ﬁne-tuned models, then he or she can do so by commenting our line 60 in the batch.py ﬁle
Lastly, all results are stored into the results/logs folder by default and can be compiled easily into a csv ﬁle using the log extractor.py script.
A. D ETAILED F UTURE W ORKS
Even though we support our main thesis with extensive empirical evidence and analyze this phenomenon through the lens of texture bias and inﬂuence functions, why, when, and how robust models transfer better deserves further investigation.
In this section we’d like to provide some ideas that we hope will spark research interest.
Different adversarial training constraint type.
Prior work only considers (cid:96) 2 and (cid:96) ∞ adversarial constraint types.
However, different adversarial constraints could allow models to retain more transferable features from the source dataset.
Two possibilities are (i) constraining on the Fischer Kernel with inﬂuence functions, and (ii) constraining on the Fourier space instead of the pixel space.
For (i), as shown by Koh & Liang (2017) for the i th image x i we can compute the perturbation δ i at each step of PGD by starting with δ i = 0 , and then δ i =
P
( δ i + α sign
I pert,loss ( x i
+ δ i , x i )) .
For (ii), we could instead calculate the gradient w.r.t.
each one of the frequencies and constrain the model to only use a subset of all of its frequencies to represent the input image.
Different source datasets.
ImageNet might not be the best source dataset for two reasons.
First, we as shown by Tsipras et al.
there are many labels that overlap with each other, such as riﬂe and assault riﬂe.
Second, there are many training images containing objects from more than one label, referred to in Tsipras et al.
as multi-object images.
Thus, we think it would be worthwhile to use Tencent’s Large-Scale Multi-Label Image Database from Wu et al.
(2019) as a source dataset instead of ImageNet.
Decision-boundary bias.
In line with Section 5 , it might be worth looking at the transferability of robust models as a function of how closely related are the labels in the target dataset.
Our hypothesis is that if the labels in the target dataset are closely related, then the robust model might transfer slightly worse than if the labels were further apart from each other.
Although measuring the closeness of labels within a dataset is challenging, this could be an interesting extension to Santurkar et al. Bias reduction and fairness.
As shown in Section 5, robust models are biased towards low resolutions and low frequencies.
Thus, it’s possible that robust models have a lower facial recognition bias than naturally trained models.
[Table]
CIFAR27.
30.17 CIFAR74.
75.95 SVHN 49.
70.19 FMNIST 82.
84.59 KMNIST  79.
80.68 MNIST 91.
CIFAR1, 37.
40.82 CIFAR1, 79.
81.1 SVHN 1, 62.
76.77 FMNIST 1, 85.
86.91 KMNIST 1, 74.
85.52 MNIST 1, 94.
CIFAR3, 51.
51.34 CIFAR3, 83.
85.07 SVHN 3, 74.
84.4 FMNIST 3, 87.
88.89 KMNIST 3, 81.
89.14 MNIST 3, 96.
CIFAR6, 62.
60.75 CIFAR6, 87.
88.6 SVHN 6, 82.
89.09 FMNIST 6, 89.
90.11 KMNIST 6, 86.
92.86 MNIST 6, 97.
CIFAR12, 69.
67.16 CIFAR12, 91.
91.6 SVHN 12, 88.
91.81 FMNIST 12, 91.
91.69 KMNIST 12, 94.
95.9 MNIST 12, 98.
CIFAR25, 75.
71.52 CIFAR25, 93.
93.28 SVHN 25, 92.
93.91 FMNIST 25, 93.
92.79 KMNIST 25, 97.
97.44 MNIST 25, 99.
CIFAR50, 78.
76.06 CIFAR50, 95.
94.58 FMNIST 60, 94.
93.98 KMNIST 60, 98.
98.38 MNIST 60, 99.
99.39 SVHN 73, 95.