What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?
Anonymous Author(s)
AffiliationAddress email
Abstract
Adversarial vulnerability of neural nets, and subsequent techniques to create robustmodels have attracted significant attention; yet we still lack a full understanding of this phenomenon.
Here, we study adversarial examples of trained neuralnetworks through analytical tools afforded by recent theory advances connectingneural networks and kernel methods, namely the Neural Tangent Kernel (NTK),following a growing body of work that leverages the NTK approximation to successfully analyze important deep learning phenomena and design algorithms fornew applications.
We show how NTKs allow to generate adversarial examples in a“training-free” fashion, and demonstrate that they transfer to fool their finite-widthneural net counterparts in the "lazy" regime.
We leverage this connection to providean alternative view on robust and non-robust features, which have been suggestedto underlie the adversarial brittleness of neural nets.
Specifically, we define andstudy features induced by the eigendecomposition of the associated kernel to betterunderstand the role of robust and non-robust features, the reliance on both forstandard classification and the robustness-accuracy trade-off.
We find that suchfeatures are surprisingly consistent across architectures, and that robust featurestend to correspond to the largest eigenvalues of the model, and thus are learnedearly during training.
Our framework allows us to identify and visualize non-robustyet useful features.
Finally, we shed light on the robustness mechanism underlyingadversarial training of neural nets used in practice: quantifying the evolution ofthe associated empirical NTK, we demonstrate that its dynamics falls much earlierinto the “lazy” kernel regime and manifests a much stronger form of the wellknown bias to prioritize learning features within the top eigenspaces of the kernel,compared to standard training.
Introduction
Despite the tremendous success of deep neural networks in many computer vision and languagemodeling tasks, as well as in scientific discoveries, their properties and the reasons for their successare still poorly understood.
Focusing on computer vision, a particularly surprising phenomenonevidencing that those machines drift away from how humans perform image recognition is thepresence of adversarial examples , images that are almost identical to the original ones, yet aremisclassified by otherwise accurate models.
Since their discovery [41], a vast amount of work has been devoted to understanding the sources ofadversarial examples and explanations include, but are not limited to, the close to linear operatingmode of neural nets [21], the curse of dimensionality carried by the input space [21, 39], insufficientmodel capacity [31, 43] or spurious correlations found in common datasets [23].
In particular,one widespread viewpoint is that adversarial vulnerability is the result of a model’s sensitivity toimperceptible yet well-generalizing features in the data, so called useful non-robust features, giving rise to a trade-off between accuracy and robustness [43, 45].
This gradual understanding has enabledthe design of training algorithms, that provide convincing, yet partial, remedies to the problem; themost prominent of them being adversarial training and its many variants [16, 21, 30].
Yet we are farfrom a mature, unified theory of robustness that is powerful enough to universally guide engineeringchoices or defense mechanisms.
In this work, we aim to get a deeper understanding of adversarial robustness (or lack thereof) byfocusing on the recently established connection of neural networks with kernel machines.
Infinitelywide neural networks, trained via gradient descent with infinitesimal learning rate, provably becomekernel machines with a data-independent, but architecture dependent kernel - its Neural Tangent
Kernel (NTK) - that remains constant during training [4, 24, 27, 28].
The analytical tools afforded bythe rich theory of kernels have resulted in progress in understanding the optimization landscape andgeneralization capabilities of neural networks [3, 17], together with the discovery of interesting deeplearning phenomena [18, 34], while also inspiring practical advances in diverse areas of applicationssuch as the design of better classifiers [38], efficient neural architecture search [14], low-dimensionaltasks in graphics [42] and dataset distillation [32].
While the NTK approximation is increasinglyutlilized, even for finite width neural nets, little is known about the adversarial robustness propertiesof these infinitely wide models.
Our contribution: Our work inscribes itself into the quest to leverage analytical tools afforded bykernel methods, in particular spectral analysis, to track properties of interest in the associated neuralnets, in this case as they pertain to robustness.
We study adversarial perturbations and robustnessof kernels so as to enrich our understanding of adversarial robustness in general machine learningmodels.
To this end, we first demonstrate that adversarial perturbations generated analytically withthe NTK can successfully lead the associated trained wide neural networks (in the kernel-regime)to misclassify, thus allowing kernels to faithfully predict the lack of robustness of those trainedneural networks.
In other words, adversarial (non-) robustness transfers from kernels to networks;and adversarial perturbations generated via kernels resemble those generated by the correspondingtrained networks.
One implication of this transferability is that we can analytically devise adversarialexamples that do not require access to the trained model and in particular its weights; instead these“blind spots” may be calculated a-priori, before training starts.
Although similar transferability hasalready been known and exploited in prior work that trains substitute models, we demonstrate that thisnotion holds from first principles that only require the a-priori description of the model architecture.
The analytical expressions afforded by the kernel might provide a better understanding of the elusiveconcept of transferability also across architectures, as the corresponding expressions for the associatedkernels can be compared directly.
A perhaps even more crucial implication of the NTK approach to robustness relates to the understanding of adversarial examples.
Indeed, we show how the spectrum of the NTK provides an alternativeway to define features of the model, to classify them according to their robustness and usefulnessand visually inspect them via their contribution to the adversarial perturbation (see Fig. 1).
Thisin turn allows us to verify previously conjectured properties of standard classifiers; dependence onboth robust and non-robust features in the data [43], and tradeoff of accuracy and robustness duringtraining.
In particular we observe that features tend to be rather invariable across architectures, andthat robust features tend to correspond to the top of the eigenspectrum (see Fig. 2), and as such arelearned first by the corresponding wide nets [3, 24].
Moreover, we are able to visualize useful nonrobust features of standard models (Fig. 5).
While this conceptual feature distinction has been highlyinfluential in recent works that study the robustness of deep neural networks [1, 25, 40], to the best ofour knowledge, none of them has explicitly demonstrated the dependence of networks on such featurefunctions (except for simple linear models [20]).
Rather, these works either reveal such features insome indirect fashion, or accept their existence as an assumption.
Here, we show that Neural Tangent e
i g
FC
FC FC
FC FC
Conv Conv e
i ge i ge
i ge i g0.
U s e f u l n e ss
NTK Features (CIFAR10, FC2)
[Figure]
s e f
[Figure]
n
[Figure]
e i g
e n v e c t
o r i n d e x
[Figure]
e i g
e n v e c t
o r i n d e
[Figure] [Figure]
0 std fgsm pgd
MNIST 25.0 30.0 35.0 40.0 45.0 50.0 55.
1.0 std fgsm pgd
CIFAR
Kernel theory endows us with a natural definition of features through its eigen-decomposition andprovides a way to visualise and inspect robust and non-robust features directly on the function spaceof trained neural networks.
Interestingly, this connection also enables us to empirically demonstrate that robust features ofstandard models alone are not enough for robust classification.
Aiming to understand, then, whatmakes robust models robust, we track the evolution of the data-dependent empirical NTK duringadversarial training of neural networks used in practice.
Prior experimental work has found thatnetworks with non-trivial width to depth ratio which are trained with large learning rates, depart fromthe NTK regime and fall in the so-called “rich feature” regime, where the NTK changes substantiallyduring training [7, 18, 19, 34].
In our work, which to the best of our knowledge is the first toprovide insights on how the kernel behaves during adversarial training, we find that the NTK evolvesmuch faster compared to standard training, simultaneously both changing its features and assigningmore importance to the more robust ones, giving direct insight into the mechanism at play duringadversarial training (see Fig. 3).
In summary, the contributions of our work are the following:
• We discuss how to generate adversarial examples for infinitely-wide neural networks via the
NTK, and show that they transfer to fool their associated (finite width) nets in the appropriate
[Figure]
regime, yielding a "training-free" attack without need to access model weights (Sec. 3).
• We then turn to the kernel
machines corresponding to infinitely-wide networks trainedwith small learning rate to deepen our understanding of adversarial robustness.
Using thespectrum of the NTK, we give an alternative definition of features, providing a naturaldecomposition or perturbations into robust and non-robust parts [23, 43] (Fig. 1).
We confirm that robust features overwhelmingly correspond to the top part of the eigenspectrum; hence they are learned early on in training.
We bolster previously conjectured hypotheses that prediction relies on both robust and non-robust features and that robustness is traded for accuracy during standard training.
Further, we show that only utilizing the robust features of standard models is not sufficient for robust classification (Sec. 4).
• We turn to finite width neural nets with standard parameters to study the dynamics of theirempirical NTK during adversarial training .
We show that the kernel rotates in a way thatenables both new (robust) feature learning and that drastically increases of the importance(relative weight) of the robust features over the non-robust ones.
We further highlightthe structural differences of the kernel change during adversarial training versus standardtraining and observe that the kernel seems to enter the “lazy” regime much faster (Sec. 5).
Collectively, our findings may help explain many phenomena present in the adversarial ML literatureand further elucidate both the vulnerability of standard models and the robustness of adversariallytrained ones.
We provide code to visualize features induced by kernels, giving a unique and principledway to inspect features induced by standardly trained nets.
Related work:
To the best of our knowledge the only prior work that leverages NTK theory to deriveperturbations in some adversarial setting is due to [44], yet with entirely different focus.
It dealswith what is coined generalization attacks : the process of altering the training data distribution toprevent models to generalise on clean data.
[6] study aspects of robust models through their linearizedsub-networks, but do not leverage NTKs.
Our work is the first to provide a kernel-induced notion offeatures and to study robustness in adversarial training in the NTK regime.
Preliminaries
We introduce background material and definitions important to our analysis.
Here, we restrictourselves to binary classification and scalar kernels, to keep notation light.
We defer the multiclasscase, complete definitions and a more detailed discussion of prior work to the Appendix.
2.1 Adversarial Examples
Let f be a classifier, x be an input (e.g. a natural image) and y its label (e.g. the image class).
Then,given that f is an accurate classifier on x , ˜ x is an adversarial example [41] for f if 1) their distanced ( x , ˜ x ) is small.
Common choices in computer vision are the ℓ p norms, especially the ℓ ∞ norm onwhich we focus henceforth, and 2) f
(˜ x ) ̸ = y .
That is, the perturbed input is being misclassified.
Given a loss function L , such as cross-entropy, one can construct an adversarial example ˜ x
= x + ηby finding the perturbation η that produces the maximal increase of the loss, solving
[Equation]
for some ϵ > 0 that quantifies the dissimilarity between the two examples.
In general, this is anon-convex problem and one can resort to first order methods [21]
[Equation]
or iterative versions for solving it [26, 30].
The former method is usually called Fast Gradient Sign
Method (FGSM) and the latter Projected Gradient Descent (PGD) .
These methods are able to produceexamples that are being misclassified by common neural networks with a probability that approaches1 [12].
Even more surprisingly, it has been observed that adversarial examples crafted to “fool” onemachine learning model are consistently capable of “fooling” others [35, 36], a phenomenon thatis known as the transferability of adversarial examples.
Finally, adversarial training refers to thealteration of the training procedure to include adversarial samples for teaching the model to be robust[21, 30] and empirically holds as the strongest defense against adversarial examples [30, 45].
2.2 Robust and Non-Robust features
Despite a vast amount of research, the reasons behind the existence of adversarial examples arenot perfectly clear.
A line of work has argued that a central reason is the presence of robust andnon-robust features in the data that standard models learn to rely upon [23, 43].
In particular itis conjectured that reliance on useful but non-robust features during training is responsible for thebrittleness of neural nets.
Here, we slightly adapt the feature definitions of [23] 1 , and extend them tomulti-class problems (see Appendix).
Let D be the data generating distribution with x ∈ X and y ∈ {± 1 } .
We define a feature as a functionϕ : X → R and distinguish how they perform as classifiers.
Fix ρ, γ ≥ 0 :
ρ -
Useful feature: A feature ϕ is called ρ -useful if
[Equation]
γ -Robust feature: A feature ϕ is called γ -robust if it remains useful under any perturbationinside a bounded “ball” B , that is if
[Equation]
In general, a feature adds predictive value if it gives an advantage above guessing the most likelylabel, i.e. ρ > max y ′ ∈{± 1 } E x,y ∼D
[ I { y ′ =
y } ] , and we will speak of "useful" features in this case,omitting the ρ .
We will call such a feature useful, non-robust if it is useful, but γ -robust only forγ = 0 or very close to 0 , depending on context.
The vast majority of works imagines features as being induced by the activations of neurons inthe net, most commonly those of the penultimate layer ( representation-layer features), but thisformal definition is in no way restricted to activations, and we will show how to exploit it usingthe eigenspectrum of the NTK.
In particular, in Sec. 4, we demonstrate that the above frameworkagrees perfectly with features induced by the eigenspectrum of the NTK of a network, providing anatural way to decompose the predictions of the NTK into such feature functions.
In particular wecan identify robust, useful, and, indeed, useful non-robust features.
2.3 Neural Tangent Kernel
Let f : R d → R be a (scalar) neural network with a linear final layer parameterized by a set ofweights w ∈ R p and {X , Y} be a dataset of size n , with X ∈ R n × d and Y ∈ {± 1 } n .
Linearizedtraining methods study the first order approximation
[Equation]
The network gradient ∇ w f ( x ; w t ) induces a kernel function Θ t : R d × R d → R , usually referred asthe Neural Tangent Kernel (NTK) of the model
[Equation]
This kernel describes the dynamics with infinitesimal learning rate (gradient flow).
In general, thetangent space spanned by the ∇ w f ( x ; w t ) twists substantially during training, and learning with the
Gram matrix of Eq.
(6) (empirical NTK) corresponds to training along an intermediate tangent plane.
Remarkably, however, in the infinite width limit with appropriate initialization and low learning rate,it has been shown that f becomes a linear function of the parameters [24, 28], and the NTK remainsconstant ( Θ t = Θ
=: Θ ).
Then, for learning with ℓ 2 loss the training dynamics of infinitely widenetworks admits a closed form solution corresponding to kernel regression [4, 24, 27]
[Equation]
where x ∈ R
d is any input (training or testing), t denotes the time evolution of gradient descent,λ is the (small) learning rate and, slightly abusing notation, Θ( X , X ) ∈ R n × n denotes the matrixcontaining the pairwise training values of the NTK, Θ( X , X ) ij =
Θ( x i , x j ) , and similarly forΘ( x , X ) ∈ R n .
To be precise, Eq.
(7) gives the mean output of the network using a weightindependent kernel with variance depending on the initialization 2 .
Transfer results in the kernel regime
In this section, we show how to generate adversarial examples from NTKs and discuss their similarityto the ones generated by the actual networks.
Note that for network results, we restrict ourselves towide networks initialized in the "lazy" regime with small learning rates (the "kernel regime").
3.1 Generation of Adversarial Examples for Infinitely Wide Neural Networks
Adversarial examples arise in the context of classification , while the NTK learning process isdescribed by a regression as in Eq.
The arguably simplest way to align with the frameworkpresented in Eq.
(1) is to treat the outputs of the kernel similar to logits of a neural net, mapping themto a probability distribution via the sigmoid/softmax function and apply cross-entropy loss.
A simple calculation (see Appendix, together with the generalization to the multi-class case) gives:
The optimal one step adversarial example of a scalar, infinitely wide, neural network is given by
[Equation]
x )) , (8) for ∥ ˜ x − x ∥ ∞ ≤ ε , where ∇ x f t ( x ) = ∇ x Θ( x , X ) ⊤ Θ − 1 ( X , X )
( I − e − λ Θ( X , X ) t ) Y .
One can conceive other ways to generate adversarial perturbations for the kernel, either by changingthe loss function (as previously done in neural networks (e.g. [12])) or through a Taylor expansionaround the test input, and we present such alternative derivations in the Appendix.
However, inpractice we observe little difference between that approach and the one presented here.
3.2 Transfer results and kernel attacks
Predictions from NTK theory for infinitely wide neural networks have been used successfully for theirlarge finite width counterparts, so it seems reasonable to conjecture that adversarial perturbationsgenerated via the kernel as in Eq.
(8) strongly resemble those directly computed for the correspondingneural net as per Eq.
In particular, this would imply that adversarial perturbations derived fromthe NTK should not only fool the kernel machine itself, but also lead wide neural nets to misclassify.
While similar transfer results in different contexts have been observed indirectly, via the effects ofthe perturbation on metrics like accuracy [32, 44], we aim to look deeper to compare perturbationsdirectly .
High similarity would imply that any gradient based white-box attack on the neural net canbe successfully mimicked by a “black-box” kernel derived attack.
[Figure]
Setting .
To this end, we train multiple two-layer neural networks onimage classifications tasks extracted from MNIST and CIFAR-10and compare adversarial examples generated by Eqs.
(2) (attackingthe neural network) and (8) (attacking the kernel).
The networksare trained with small learning rate and are sufficiently large, so lieclose to the NTK regime.
We track cosine similarity between thegradients of the loss from the NTK predictions and the gradientsfrom the actual neural net as training evolves.
Then, we generateadversarial perturbations from both the neural net and the kernelmachine, and test whether those produced by the latter can fool theformer.
Full experimental details can be found in the Appendix.
Results .
Our experiments confirm a very strong alignment of lossgradients from the neural nets and the NTK across the whole duration of training, as can be seen in Fig.
4 (top).
Then, as expected,kernel-generated attacks produce a similar drop in accuracy throughout training as the networks “own” white-box attacks, eventuallydriving robust accuracy to 0% , as seen in Fig.
4 (bottom).
We reproduce these plots for MNIST in the Appendix, leading to similarconclusions.
When concerned with security aspects of neural nets, adversarialattacks are mainly characterised as either white-box or black-boxattacks [36].
White box attacks assume full access to the neuralnetwork and in particular its weights; prominent examples include FGSM/PGD attacks.
Black boxattacks, on the other hand, can only query the model to try to infer the loss gradient, either throughtraining separate surrogate models [35] or through carefully crafted input-output pairs fed to thetarget model [2, 13, 22].
NTK theory and the experiments of this section suggest a threat model inwhich the attacker does not require access to the model or its weights, nor training of a substitutemodel.
For fixed architecture and training data, all the information required for the computation of(8) is available at initialization, making the "NTK-attack" akin to a "training free" substitution attack,and, at least in the kernel-regime for wide nets considered here, as effective as white-box attacks.
NTK eigenvectors induce robust and non-robust features
This close connection between adversarial perturbations from the kernel and the corresponding neuralnet gives us the opportunity to bring to bear kernel tools on the study of adversarial robustness and itsrelation to features in a more direct fashion.
Several recent works leverage properties of the NTK,and specifically its spectrum, to study aspects of approximation and generalization in neural networks[3, 8, 9, 10].
Here we show how the spectrum relates to robustness and helps to clarify the notion ofrobust/non-robust features.
We define features induced by the eigendecomposition of the Gram matrix
Θ( X , X ) =
(cid:80) ni
=
λ i v i v ⊤ i .
We will be most interested in the end of training, when the model has access to all the features it canextract from the training data X .
As t → ∞ , Eq.
(7) becomes f ∞ ( x ) =
Θ( x , X ) ⊤ Θ
( X , X ) − 1 Yand can be decomposed as f ∞ ( x )
=
Θ( x , X ) ⊤ (cid:80) ni
=1 λ − 1 i v i v ⊤ i Y =
(cid:80) ni
=1 f ( i ) ( x ) , where
[Equation]
Each f ( i ) can be seen as a unique feature captured from the (training) data.
Note that these functionsmap the input to the output space, thus matching the definitions of Sec. 2.2.
Also observe thatall f
( i ) ’s jointly recover the original prediction of the model, while each one, intuitively, shouldcontribute something different to it.
Importantly, these features induce a decomposition of the gradient of the loss into parts, eachrepresenting gradients of a unique feature as already advertised in Fig.
The binary case isparticularly elegant as it gives rise to a linear decomposition of the gradient as
[Equation]
= α i ∇ x L ( f ( i ) ( x ) , y ) , (10) for some α i depending on x and y
(see Appendix).
But if f ( i ) ’s are features, how do they look like?
[Figure]
tures earlier and later
Feature properties of common architectures: With thesedefinitions in place, we can now analyze the characteristicsof features for commonly used architectures, leveraging theirassociated NTK.
To be consistent with the previous section, weconsider classification problems from MNIST (10 classes) and
CIFAR-10 (car vs airplane).
We compose the Gram matricesfrom the whole training dataset (50000 and 10000, respectively),and compute the different feature functions f
( i ) using the eigendecomposition of the matrix.
We estimate the usefulness of afeature f
( i ) by measuring its accuracy on a hold-out validationset, and its robustness by perturbing each input of this set,using an FGSM attack on feature f ( i ) .
We consider severaldifferent Fully Connected and Convolutional Kernels, whoseexpressions are available through the Neural Tangents library[33], built on top of JAX [11].
We summarize our findings onhow these features behave:
Functions f
( i ) represent visually distinct features.
We visualise each feature f
( i ) by plotting itsgradient with respect to x .
Fig.
2 shows the gradient of the first 5 features for various architecturesfor a specific image from the CIFAR-10 dataset.
We observe that features are fairly consistent acrossmodels, and they are interpretable: for example the 4th feature seems to represent the dominant colorof an image, while the 5th one seems to be capturing horizontal edges.
Networks use both robust and non-robust features for prediction.
It has been speculated that neuralnetworks trained in a standard (non adversarial) fashion rely on both robust and non-robust features.
Our feature definition in Eq.
(9) shows that this is indeed the case.
The NTK of common neuralnetworks consists of both robust features that match human expectations, such as the ones depicted in
Fig. 2, but also on features that are predictive of the true label, while not being robust to adversarialperturbations of the input (Fig. 5).
Fig.
2 depicts the first 100 features of a fully connected and aconvolutional tangent kernel in Usefulness-Robustness space.
The upper left region of the plots showsa large amount of useful, yet non-robust features.
These features seem random to human observers.
[Figure]
Robustness lies at the top.
We observe in Fig.
2 that features corresponding to the top eigenvectorstend to be robust.
This is consistent among different models and between the two datasets (see
Appendix).
Since these eigenvectors are the ones fitted first during training [3, 24], it is no wonderthat the loss gradient evolves from coherence to noise, as observed in Fig.
A1.
This also explains theapparent trade-off between robustness and accuracy of neural networks as training progresses: useful,robust features are fitted first, followed by useful, but non-robust ones.
This ties in well with bothempirical findings [37] and theoretical case studies [8, 9, 10] that demonstrate that low frequencyfunctions are fitted first during training and provide favorable generalization properties and we wouldassociate robust features with these low-frequency parts.
Robust features alone are not enough.
In light of these findings, it might be reasonable to conjecturethat we could obtain robust models by retaining the robust features of the prediction, while discardingthe non-robust ones.
The spectral approach gives a principled way to disentangle features and createkernel machines keeping only the robust ones.
Our results show that in general it is not possible toobtain non-trivial performance without compromising robustness in this fashion, strengthening thecase for the necessity of data augmentation in the form of adversarial training (see the Appendix foran in-depth study).
Kernel dynamics during adversarial training
Given the apparent necessity for adversarial training to produce robust models, how does it achievethis goal?
To shed some light on this fundamental question, we depart from the “lazy” NTK regimeand study the evolution of the NTK of adversarially trained models.
For a neural network trainedwith gradient descent, as the learning rate η → 0 , the continuous time dynamics can be written as
[Equation]
In the NTK regime, this kernel Θ t remains fixed at its initial value.
However, outside this regime, ithas been demonstrated, both empirically [7, 18, 19, 34] and theoretically [5], that Θ t is not constantduring training, and is changing as the weights move.
In adversarial training, moreover, there is theadditional effect that at each weight update, the data changes as well.
For that reason, understandingthe dynamics of adversarial training requires tracking the evolution of a kernel Θ t ( X t , X t ) , where
X t denotes the current (mini) batch of training data.
Notice that the tangent vector ∇ w f ( X t ) is stilldescribing the instantaneous change of f on the current batch of data, thus Θ t ( X t , X t ) is informativeof the local geometry of the function space, justifying its value as a quantity to be measured duringadversarial training.
We train a deep convolutional architecture on CIFAR-10 (multiclass) with standard (sgd) and adversarial training using PGD with an ℓ ∞ constraint.
Full implementations details and accuracy curvescan be found in the Appendix, together with the reproduction of the same experiment on MNIST,where the observations are similar.
We track the following quantities during training:
Kernel distance.
We compare two kernels using a scale invariant distance , which quantifies the relative rotation between them, as used in other works studying NTK dynamics (e.g. [18]):
[Equation]
Polar dynamics .
Zooming in on the change that the initial kernel undergoes, we define a polar spaceon which we measure the movement of the kernel:
[Equation]
(12) where Θ 0 , Θ f are the initial and final kernel, respectively.
Fig.
3 presents a heatmap of kerneldistances at different time steps for both standard and adversarial training, as well as both trainingtrajectories in polar space.
Concentration on subspaces.
To quantify weight concentration on the top region of the spectrum,we track the (normalized) Frobenius norm of subspaces as (cid:80) pi =1 λ 2 i / (cid:80) ni
=
λ 2 i , for various cut-offs p ,where we have indexed the eigenvalues from largest to smallest.
Fig.
6 depicts concentration on thetop 20 eigenvalues during training.
Our findings show that similar to what has been reported in prior work [18], the kernel rotatessignificantly in the beginning of training and then slows down for both standard and adversarialtraining.
However, in the latter case, this second phase begins a lot earlier.
As Fig.
3 illuminates,the kernel moves a greater distance than when performing standard training, but after a few epochsit stops both rotating and expanding; note that this is not the case for standard training where thekernel increases its magnitude substantially later in training, and in fact grows to have a norm ordersof magnitude larger than during adversarial training (see Fig. 6).
In hindsight, this behavior isperhaps not surprising, as each element of the kernel measures similarity between data points, anda robust machine should be more conservative when estimating similarity.
The observation thatduring adversarial training the kernel becomes relatively static relatively fast might indicate thatlinear dynamics govern the later phase of adversarial training.
It has been observed in previousworks [18, 19, 34] that linearization after a few initial epochs of rapid rotation often closely matchesperformance of full network training.
Our results indicate that a similar phenomenon occurs evenunder the data shift of adversarial training, opening avenues to design robust machines more efficiently.
Moreover, endowed with the knowledge that at least for kernels trained with static data robust featureslie at the top, we study polar dynamics of the top space only (see Appendix) to observe that there issubstantial rotation in this space, suggesting that robust features are learned early on not only duringstandard, but in particular during adversarial training.
Even more interestingly, Fig.
6 demonstratesthat not only the robust features change, but their relative weight as measured by the concentration onthe top-20 space is increasing simultaneously relative to standard training as well, and remains large;in fact, significantly larger than during standard training.
As each eigenvalue weights the importanceof the corresponding feature on the final prediction, this implies that the kernel “learns” to dependmore on the most robust features.
Put together, these findings reveal different kernel dynamics during standard and adversarial training:the kernel rotates much faster, expands much less and becomes “lazy” much earlier than duringstandard training.
Fully understanding the properties of converged adversarial kernels remains animportant avenue for future work, that might allow to design faster algorithms for robust classification.
Final Remarks
We have studied adversarial robustness through the lens of the NTK across multiple architecturesand data sets both in the idealized NTK regime and the “rich feature” regime.
When connecting thespectrum of the kernel with fundamental properties characterizing robustness our phenomenologicalstudy reveals a universal picture of the emergence of robust and non-robust features and their roleduring training.
There are certain limitations and unexplored themes in our work; Sec. 3 argues thattransferable attacks from the NTK may be as effective as white-box attacks, but this warrants anin-depth study across architectures, kernels and data sets (which has not been the main focus of thiswork).
Sec. 4 visualises features for fairly simple models, since the computation of kernel derivativesis a costly procedure.
It would be interesting to use our framework to visualise features from morecomplicated architectures.
Finally, our work in Sec.
5 invites more research on the kernel at the endof adversarial training, similar to what has been done for standard models [29].
We hope that our viewpoint can motivate further theoretical understanding of adversarial phenomena(such as transferability) and the design of better and/or faster adversarial learning algorithms, byfurther analyzing the kernels from robust deep neural networks.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 4 and Appendix.•
Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]Please
do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section
does
not count towards the page limit
In your paper, please delete this instructions
block and
only keep the Checklist section
heading above along
with the
questions
/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Our work
sheds light properties of adversarial examples to make mahcine learning models more
reliable in the long run.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical
[Table]
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]