Understanding Robust Learning through the Lens of Representation Similarities
Anonymous Author(s
AffiliationAddress
Abstract
Representation learning, i.e. the generation of representations useful for downstream applications, is a task of fundamental importance that underlies much ofthe success of deep neural networks (DNNs).
Recently, robustness to adversarialexamples has emerged as a desirable property for DNNs, spurring the developmentof robust training methods that account for adversarial examples.
In this paper, weaim to understand how the properties of representations learned by robust trainingdiffer from those obtained from standard, non-robust training.
This is critical todiagnosing numerous salient pitfalls in robust networks, such as, degradation ofperformance on benign inputs, poor generalization of robustness, and increase inover-fitting.
We utilize a powerful set of tools known as representation similaritymetrics, across three vision datasets, to obtain layer-wise comparisons betweenrobust and non-robust DNNs with different architectures, training procedures andadversarial constraints.
Our experiments highlight hitherto unseen properties ofrobust representations that we posit underlie the behavioral differences of robustnetworks.
We discover a lack of specialization in robust networks’ representationsalong with a disappearance of ‘block structure’.
We also find overfitting duringrobust training largely impacts deeper layers.
These, along with other findings,suggest ways forward for the design and training of better robust networks.
Introduction
Representation learning is fundamental to machine learning and a task on which deep neural networksperform remarkably well.
Given complex high-dimensional input signals, such as images, speechor text, deep neural networks (DNNs) can learn meaningful, lower-dimensional representations [22,6, 37, 11, 17, 10, 44].
The downstream utility of these representations underlies DNNs’ successon different tasks [5, 13, 33, 24, 3].
For standard image classification, neural networks learn wellseparated representations of inputs from different classes to achieve high accuracy.
The discoveryof adversarial examples, perturbed inputs that induce high error in model predictions, even for welltrained classifiers has motivated the need for another desirable property: robustness .
Robust trainingmethods typically protect against adversarial examples during the training phase by converting thestandard empirical risk minimization objective to a min-max one (Figure 1) [23, 32, 43].
However,they typically have far worse performance on benign inputs, converge slower and appear to requirelarger architectures to be effective [14, 26, 36, 39].
Given the importance of representation learning forgood downstream performance, we posit that an effective method to identify potential improvementsfor robust training is via developing a better understanding of robustly learned representations.
Our approach is to juxtapose representations of non-robust and robust networks and analyze themthrough a unified lens.
Aggregate metrics like loss and accuracy do shed some light on the differences,but these are task-driven and do not admit a layer-wise comparison of representations [1, 15].
We need a comprehensive analysis of the impact of choice of architecture, nature of threat model andcorresponding attack strength, and the choice of dataset on robustly learned representations and theirlink to downstream performance.
[Figure]
Robust training aims to learn decision boundaries that are robust to adversarial perturbations ( in sketch), naturally learning complex decision boundaries than non-robust training.
To this end, we leverage recent work on representationsimiliarity (RS) that enables fine-grained comparisons ofrepresentations obtained across all layers in a deep neuralnetwork (DNN) [20, 30].
Our key findings, using both benign and adversarial inputs as probes over CIFAR-10 [21]and the Imagenette and Imagewoof subsets of Imagenet[18] dataset, are:
Robust representations are less specialized (§3): Robust networks exhibit much higher similarity across spatially distant layers compared to non-robust networks,which have far more localized similarity.
The latter thusexhibit the expected specialization in layers, which is unexpectedly missing in the former.
Further, we find reducingthe width of the network has a similar effect to increasing the budget used in robust training, with both changesincreasing long-range similarity among the learned representations.
This indicates that the relative capacity of anetwork drops as the budget is increased.
Early layers are largely unaffected by adversarial examples (§4): The representations of perturbed inputs from non-robust and robust networks are, as expected, starkly different.
On the otherhand, the representations of benign and perturbed inputs from robust networks are indistinguishablefrom one another with regards to representation similarity metrics.
Deeper layers overfit (§5):
For both non-robust and robust training, early layers in the networkconverge within the first few epochs, and exhibit very little variation in subsequent training.
Thisindicates they perform similar roles in both benign and robust networks.
Deeper layers convergeslower while visiting drastically different representations over the course of training.
These differentrepresentations usually occur when the loss indicates overfitting, leading us to conclude that deeperlayers overfit during robust training .
Visually dissimilar adversarial constraints can lead to similar robust representations (§6):
Regardless of the adversarial constraint, we find a high degree of similarity between the representationsof perturbed and benign inputs for robust networks.
Representations from networks robust to different,but allied ( e.g. ℓ p ), threat models are similar across attack strength.
We find surprising similaritiesbetween the representations of visually unrelated threat models at lower attack strengths.
What do our findings imply?
The lack of specialization in robust networks even with heavilyoverparametrized architectures suggests a need for explicit tools to differentiate between layers duringtraining.
Further, the overfitting we find in deeper layers of neural networks is an indication thatlayer-wise specialized regularization methods need to be adopted for robust training.
We hope thelessons from this paper and the accompanying open-source code spur the development of robustlearning-specific architectures and training methods.
Paper Layout.
We provide an overview of RS tools and robust training methods in §2.
We comparerepresentations from non-robust and robust networks using benign inputs in §3.
§4 repeats thisanalysis with perturbed inputs.
The evolution of robust representations over training is studied in§5 while the impact of different threat models is in §6.
§7 concludes with lessons learned from theanalysis.
Background and Setup
We provide a brief overview of the techniques we use to train robust networks and the tools we use toprobe the intermediate representations of neural networks.
Further related work is in Appendix A.
2.1 Adversarial Examples and Robust Learning
Adversarial examples are perturbed inputs to machine learning models that intentionally inducemisclassification [4, 34].
For an input x , an untargeted adversarial example with respect to a model fand loss function ℓ ( · ) is generated by adding a perturbation δ such that δ = arg max x + δ ∈ N ( x ) ℓ ( f ( x +δ ) , y ) , where y is the label of the original class and N ( · ) represents the adversarial constraint orneighborhood.
The most commonly used constraints are L p -ball based, although we also experimentwith other unrelated classes of attacks, such as JPEG, Gabor, or Snow based perturbations [19].
We characterize activations of internal layer in a network as: 1) Benign representation: if input isnon-adversarial 2) Adversarial representations: if input is an adversarial example.
[Table]
Adversarial training [23] has emerged as the most effectivemethods to train robust classifiers [7], where it accounts foradversarial examples during training.
It modifies the empirical risk minimization procedure in non-robust trainingto a min-max optimization, where the inner maximizationfocuses on finding the strongest adversarial example foreach input during training:
[Equation]
where D is a data distribution and f θ is a model with parameters θ .
Newer methods, like TRADES[43], attempt to improve on adversarial training by better balancing the tradeoff between robustnessand accuracy.
We refer to models trained using methods that account for adversarial examples asrobust models and those trained without these methods as non-robust models.
We refer to accuracy on non-perturbed input as benign accuracy and on adversarial examples asrobust accuracy .
Adversarial training is well known to trade-off benign accuracy to gain robustnessand exhibits a high generalization gap in robust accuracy (as we show in Table 1 for CIFAR-10dataset).
2.2 Probing Neural Network Representations
Representation similarity (RS) metrics allow for the quantitative comparison of the activations ofdifferent sets of neurons on a given dataset.
Comparing learned, internal representations within andbetween neural networks of different architectures is important for understanding the performanceand behavior of these networks under changing conditions.
Several metrics have been proposed forthis purpose: Canonical Correlation Analysis (CCA) [16] and its extensions, Singular Vector CCA(SVCCA) [30] and Projection Weighted CCA (PWCCA) [25], Centered Kernel Alignment (CKA)[20], the Orthogonal Procrustes distance [12] and model stitching [2, 8].
These metrics can all handledifferent sized representations, but require that the number of datapoints be the same.
We choose CKA to measure representation similarity, due to its desirable properties such as orthogonaland isotropic invariance, and wide use in previous work [27, 28, 29, 38].
Ding et al.
[12] have shownthat CKA is nevertheless insensitive to deletions of principal components of representations thatstrongly correlate with model accuracy.
However, this property does not impact the manner in whichwe use CKA.
The Appendix contains experiments with other RS metrics, yielding similar discoveries.
Centred Kernel Alignment (CKA): Let X ∈ R n × p 1 with rows x i , and Y ∈ R n × p 2 with rows y ibe the activation matrices of layers with p 1 and p 2 neurons respectively, defined on the same set of npoints.
These activation matrices are then the representations induced by a particular layer.
We alsodefine a n × n centering matrix H =
I n − 11 ⊤ .
The CKA between these activation matrices is then
[Equation]
where K ij = k
( x i , x j ) ,
L ij = l ( y i , y j ) , with k and l being kernels.
When the kernels are linear andthe activation matrices are centered, we get the Linear CKA between activation matrices to be
[Equation]
[Figure]
We measure the similarity of benign representations across all pairs of layers in both non-robust and robust networks.
Robust networks have an amplified degree of cross-layer similarity, even at long-range, which leads to vanishing of the well known block structure characteristic observed in non-robust networks [28]
) We only observe a poor similarity between representations of benign and robust models, when experimental setup is identical for both.
Since Kornblith et al.
[20] find the use of a linear kernel to be as accurate as and much faster than the
RBF kernel, we focus only on linear kernels.
For further speed-ups, we use online CKA which usesbatching to get an unbiased estimate of CKA [28].
2.3 Experimental Setup
We provide a brief overview of our experimental setup here.
All of our experiments were run onmachines which has either NVidia Titan X or 2080Ti GPUs with 8GB of memory.
Further details arein Appendix A.
Models and datasets.
We consider three commonly used image dataset: CIFAR-10 [21], ImageNette [18], and ImageWoof [18], where latter two are subsets of ImageNet dataset [9], whichwe use due to the high cost of adversarial training on the full ImageNet dataset.
We use the Wide
Resnet [41] class of models on CIFAR-10 dataset and ImageNet-based ResNets for ImageNette and
Imagewoof datasets.
Adversarial training.
We follow standard convention with ℓ ∞ perturbations and use ϵ = 8255 for
CIFAR-10 and ϵ = 4255 for ImageNet based datasets.
We use 10-step projected gradient descent(PGD) attack during training and use 20 steps at test time to calculate adversarial representations.
Representation similarity.
We will be using mainly using CKA to compare activation matricesderived from different benign or adversarial images.
In online CKA, we use a batch-size of 1024and take 3 passes over the dataset to reduce any stochasticity in the output similarity score.
Inlayer-wise similarity analyses
, we consider all layers, including convolutions, pooling, activation, andbatch-normalization layers.
How do representations from robust and non-robust networks differ?
Robust training is expected to be a harder training objective than non-robust training, since it optimizesfor both accuracy and robustness.
The resulting trade-off is also reflected in final performance, wherethe benign accuracy of robust networks is much lower than non-robust networks [23].
In this section,we aim to understand the impact of robust training on internal representations, in particular by [Figure] comparing benign representations in non-robust and robust networks.
Further experiments and resultsfrom this section are included in Appendix B.
We take the following two-fold approach:1) Comparing characteristics:
Absence of block structure.
When comparing layerwise similarityin non-robust networks, numerous works have revealed a ‘block structure’, with high similaritybetween the layers comprising a block, and low similarity outside it [20, 28, 31] (Non-robust networkplots in Figure 2).We observe that when moving from benign to robust training, while keepingeverything else constant, the block structure all but disappears .
The lack of a block structure inrobustly trained networks is accompanied by the presence of long-range similarities between layers,with layers having high similarity to those much farther away in the network (Figure 2).
2) Direct comparison: Poor similarity between non-robust and robust network representations.
To further investigate the unique characteristic of robust networks, we conduct a cross-layer comparison of benign representations in non-robust and robust networks (Figure 2d).
We find a very lowdegree of similarity across both network representations, which further shows that robust networkshad indeed learned strikingly different representations than non-robust networks.
We also show that both of these observations generalize across architectures, datasets and threatmodels.
We now delving deeper in analyzing cross-layer characteristics of robust networks representations.
Impact of robust training strength.
We investigate this disappearance of block structure in robustlytrained networks further by varying both the adversarial budget ( ϵ ) used for training and the width ofthe network (Figure 3).
As the training ϵ increases, the block structure becomes fainter, eventuallydisappearing entirely.
The block structure is also impacted by network width, with it being marginallymore pronounced in larger networks.
However, this distinction is largely absent at larger values ofϵ .
This establishes that increasing the value of ϵ is making the task more complex, so maintainingthe same model capacity leads to a decrease in relative capacity (indicated by high similarity acrossrepresentations). [Figure] a y e r ( [Figure]
e n i g n f e a t u r
[Figure] s ) [Figure] Impact of architecture.
Previously Nguyen et al.
[28] observed that increasing network width, thuscapacity, leads to emergence of block-structure in non-robust networks.
Thus we incrementallyincrease the width of robust Wide-ResNet architecture on CIFAR-10 dataset and measure the crosslayer representation similarity of benign features (Figure 3, expanded on in App. B).
Even afterincreasing network capacity by an order of magnitude, we don’t see significant variation in thecharacteristics of robust network representations.
These results suggest that while increasing width inrobust networks doesn’t lead a drastic shift in similarity of internal layer representations.
Probing the impact of adversarial perturbations on representations
Robust training aims to learn both high quality benign and robust representations.
In the previoussection we characterized the impact of robust training on benign representations.
In this section,we focus on understanding the characteristics of adversarial representations in robust networks.
Detailed results, as well as ablations on architecture and threat model, are available in Appendix C.
We first analyze how adversarial perturbations distorts the internal representations as we go deeperinto the network, i.e., compare adversarial and benign internal representations.
Next we analyzewhether the distortions introduced by adversarial perturbations can also provide insights into whysuch perturbations transfer between different networks.
4.1 Impact of adversarial perturbations across layers in non-robust and robust networks
In this experiment, we measure the CKA similarity between benign image representations andcorresponding adversarial images representations.
A fraction of early layers are largely unaffected.
Adversarial perturbation are crafted to corruptfinal layer representations, where as expected, the similarity score decays to near zero by final layer(Figure 4).
However, we also observe another surprising trend where the adversarial representationsshares a high degree of similarity with a fraction (up to one-third for ImageNette and Imagewoofdatasets) of early layers.
We find this observation holds across architectures and budgets, indicatingthat earlier layers representations, even in benign networks, are robust to adversarial perturbations.
Robust networks successfully prevent the distortion of internal representations.
For robustlytrained networks, benign and robust representations maintain a high degree of similarity throughoutthe network, with divergence only occurring in the last 10 layers or so.
In our ablation on network size,we see that increasing the size of the network leads to increased differentiation, implying sufficientcapacity for increased separability between benign and perturbed representations.
4.2 Understanding transferability through the lens of internal representation similarity
It is well-known that adversarial examples are transferable from one architecture to another, i.e.,adversarial example generated on one network achieve the misclassification objective on others too. In Figure 5a, we measure the transferability among non-robust and robust networks. Intriguingly,
[Figure]
earlier layers but drops off towards the end.
In contrast, it drops right after the first layer from robust to non-robust network, a set of networks which also have least transferability.
as previous also validate by [7], transferability is higher among some networks (non-robust to nonrobust) while much poorer for other (robust to non-robust).
High transfer ability indicates that thetarget network has learned similar decision boundary.
We aim to better understand this phenomenonby comparing the internal representations of source and target networks.
We first generate adversarialexample, and their corresponding internal layer representations, from a source network (W RN -28-10in this case).
We compare it with internal layer representations of these adversarial examples on targetnetwork.
We consider transfer from non-robust to robust, robust to robust, and robust to non-robustnetworks 1 (Figure 5).
For transfer from non-robust to non-robust or robust-robust, where highest transferability occurs,we observe high degree of similarity between representations at early layers.
It indicates that thetransferred adversarial examples distort the internal representations in a similar fashion as sourcenetwork.
The most intriguing aspect in Figure 5a is extremely poor transferability from robustto non-robust networks.
We establish that this is due to significant differences in representationswhere, even across identical architectures, we observe a large drop in CKA right after input layer.
It shows that adversarial perturbations from robust nework distorts the internal representations ofnon-robust network in a strikingly different manner (even at very early layers), which likely rendersthem ineffective to cause an adversarial effect at the output of the network.
Evolution of internal representation dynamics during robust training
In this section, we track the evolution of representations learned by robust training methods over thecourse of training epochs in the optimization process.
We study following two convergence properties:1) Rate of convergence at different layers, 2) Stability of different layers throughout learning.
Asin earlier sections, we use CKA to measure similarity, but in this section we compare the benignrepresentations of the same layer across time .
We present our results for Wide-ResNet architecture infigure 6, where we compare the representations at each epoch to the final epoch (epoch 100).
Wefurther investigate it across architectures, different layers, and attack budgets in Appendix D.
Early layers converge faster than later layers.
Benign representations after the first two blocks of
W RN -28-10 are highly similar to the representation obtained at the end of training, after just tenepochs (Figure 6).
This is true for all the training methods considered.
In App. D, we show thislargely holds true across architectures and smaller budgets.
Our observation relates to the intuitionthat early layers learn simpler features [22, 40, 42], thus are learned much faster than complex featurelearned by later layers.
Only with high budget ( ϵ = 16255 ), i.e., stronger attacks, we observe a deviationfrom this trend where second block convergence is slower.
Overfitting is predominantly visible in later layers.
From the cross-epoch heatmap of similaritiesbetween adversarial representations (App. D), we observe that later layer representations only exhibit see [Figure] similarity off-diagonal at the beginning (until epoch 30) and end of training.
This clearly shows thatmultiple local minima are found during training.
This is true to a less pronounced degree for benignrepresentations.
In Fig.
6, we also observe the adversarial validation loss increasing after around 30epochs, while the training loss keeps decreasing, indicating overfitting.
After this point in training,the CKA similarity between drops with respect to both the epoch 30 and final representation dropssignificantly in later layers, while the earlier layers remain unaffected.
This strongly indicates thatthe later layers have overfit to the training data.
Adversarial training reduces stability of internal representations during training.
While subsequent epoch representations tend to be very similar to each other in non-robust training, therethere can be very large jumps in similarity between one epoch and the next, indicating a less stableconvergence.
This effect further accentuates when training with a higher perturbation budget or whentraining a larger network (App. D).
How does the threat model impact representations?
Throughout the paper, we have focused on a single threat model: ℓ ∞ perturbations.
In this section,we extend our analysis of robust representations to understand what exactly is the link between thevisual similarity of threat models and the representations of models trained to be robust to them.
Tothis end, we consider four additional threat models: ℓ 2 , JPEG, Gabor and Snow [19] (visualized in
App. A).
We also examine the generalizability of our earlier results across these threat models.
Forspace reasons, all associated figures from this section are in App. E.
Do visually similar attacks lead to similar robust representations?
RS gives us a method bywhich we can compare how different classes of adversarial perturbations affect learned representations.
Using CKA, we plotted the cross-layer similarity of robust WRN-28-10 networks adversarially trainedagainst five different threat models ( ℓ 2 , ℓ ∞ , JPEG, Gabor, and Snow) (Figure 7).
These comparisonslend credence to conventional knowledge about these threat models.
The layerwise similarity plotbetween l 2 and l ∞ robust models is strikingly similar to CKA plots between robust models trained onthe same threat model.
This mirrors theoretical results on correspondences between ℓ p threat models[35].
We also observe a higher average similarity when comparing an ℓ ∞ model with a JPEG modelthan with Snow or Gabor models.
This makes intuitive sense, given the ℓ ∞ constraint inherent inthe JPEG attack.
When using CKA to compare against the Snow threat model, we observe that the [Figure] highest average similarity is achieved with Gabor.
This represents a novel insight into these threatclasses, as correspondence between the two was not previously known.
Alignment of budgets across different attacks.
CKA can also give insight into the varying effectsof changing perturbation strengths within different threat models.
Expanding on our experimentsabove (all figures are in App. E), we compared representations between different threat models whilevarying perturbation strengths.
Between ℓ ∞ and Gabor models, the layerwise similarity structureis much more sensitive to changes in the strength used in ℓ ∞ -robust training than in Gabor-robusttraining.
For more closely related attacks, like ℓ ∞ and JPEG, we see similar degradations of thestructure when changing the strength of either attack.
Furthermore, the structure is largely preservedwhen the attack strengths are changed in tandem.
These results may be useful for developingtechniques for training models to be jointly-robust against different classes of attack.
Layerwise structure across threat models.
We find that across all threat models except Snow,robust and benign activations exhibit high similarity (figures in App. E).
Further, as we changebudgets across the same threat model, we find that ℓ p threat models have representations that evolvemore gradually.
Discussion
In this paper, we demonstrated the ability of representation similarity metrics to enable a betterunderstanding of robust neural networks. We identified several salient differences between internalrepresentations of non-robust and robust models.
We showed that robust models exhibit far higherinter-layer similarity and are prone to overfitting in deeper layers. These observations highlight thatrobust networks could be improved by encouraging greater differentiation between layers duringtraining as well as targeted reduction of capacity in deeper layers.
In addition, we showed whytransferability is so low from adversarial examples from robust models to non-robust ones. Ourresults on different threat models indicate a path forward for networks robust to multiple attackers aswell by allowing for the determination of budgets and layers leading to the best alignment in features.
The key limitations of our study arise from the fundamental disjunct between aggregate propertiesand layer-wise representation similarity metrics.
The latter serve mainly to highlight trends anddifferences between different models, but do not provide a direct method to improve upon aggregateproperties like loss and accuracy. These improvements have to be inferred from a careful analysis ofsimilarities, and future work needs to build upon the observations in this paper for actionable changesto training strategies and architectures.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See Section 3, 4, 5, 6(b)
Did you describe the limitations of your work?
[Yes] See Section 7(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Our paperis of an abstract nature and is quite removed from direct societal impact.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] Having read the guidelines, given the abstract nature of the paper and the use of publicly available datasets, we do not foresee any ethical concerns from the
publication of this paper.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] All claims inthe paper are of an empirical nature
(b) Did you include complete proofs of all theoretical results?
[N/A] See above3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] Code to reproduce all figures and results is included in the supplemental material
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] The baseline setup is in Section 2 and each section including theAppendix contains additional details as necessary.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[No] Our results are averaged over 3 runs with the variation being too small to appear on our plots.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Section 2 contains compute details4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Citations are included
throughout the paper as appropriate
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
All included assets are publicly available.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] See above
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] See above
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]