Understanding Robust Learning through the Lens of Representation Similarities
Anonymous Author(s)
AffiliationAddress
Abstract
Representation learning, i.e. the generation of representations useful for downstream applications, is a task of fundamental importance that underlies much ofthe success of deep neural networks (DNNs).
Recently, robustness to adversarialexamples has emerged as a desirable property for DNNs, spurring the developmentof robust training methods that account for adversarial examples.
In this paper, weaim to understand how the properties of representations learned by robust trainingdiffer from those obtained from standard, non-robust training.
This is critical todiagnosing numerous salient pitfalls in robust networks, such as, degradation ofperformance on benign inputs, poor generalization of robustness, and increase inover-fitting.
We utilize a powerful set of tools known as representation similaritymetrics, across three vision datasets, to obtain layer-wise comparisons betweenrobust and non-robust DNNs with different training procedures, architectural parameters and adversarial constraints.
Our experiments highlight hitherto unseenproperties of robust representations that we posit underlie the behavioral differences of robust networks.
We discover a lack of specialization in robust networks’representations along with a disappearance of ‘block structure’.
We also findoverfitting during robust training largely impacts deeper layers.
These, along withother findings, suggest ways forward for the design and training of better robustnetworks.
Introduction
Representation learning is fundamental to machine learning and a task on which deep neural networksperform remarkably well.
Given complex high-dimensional input signals, such as images, speech,or text, deep neural networks (DNNs) can learn meaningful, lower-dimensional representations [23,6, 39, 12, 18, 11, 47].
The downstream utility of these representations underlies DNNs’ successon different tasks [5, 14, 35, 25, 3].
For standard image classification, neural networks learn wellseparated representations of inputs from different classes to achieve high accuracy.
The discoveryof adversarial examples, perturbed inputs that induce high error in model predictions, even forwell-trained classifiers has motivated the need for another desirable property: adversarial robustness .
Robust training methods typically protect against adversarial examples during the training phase by converting the standard empirical risk minimization objective to a min-max one (Figure 1)
However, they typically have far worse performance on benign inputs, converge slowerand appear to require larger architectures to be effective [15, 27, 38, 42].
Given the importance of representation learning for good downstream performance, we posit that an effective method to identify potential improvements for robust training is via developing a better understanding ofrobustly learned representations 1 .
Our approach is to juxtapose representations of non-robust and robust networks and analyze themthrough a unified lens.
Aggregate metrics like loss and accuracy do shed some light on the differences,but these are task-driven and do not admit a layer-wise comparison of representations [1, 16].
Weneed a comprehensive analysis of the impact of choice of architectural parameters like depth and,nature of threat model and corresponding attack strength, and the choice of dataset on robustly learnedrepresentations and their link to downstream performance.
[Figure]
Adversarially robust training aims to learn decision boundaries that are robust to adversarial perturbations ( ℓ ∞ in this sketch), naturally learning more complex decision boundaries than nonrobust training.
To this end, we leverage recent work on representationsimiliarity (RS) that enables fine-grained comparisons ofrepresentations obtained across all layers in a deep neuralnetwork (DNN) [21, 31].
Our key findings, using bothbenign and adversarial inputs as probes over CIFAR-10[22] and the Imagenette and Imagewoof [19] subsets ofthe Imagenet [9] dataset, are:
Robust representations are less specialized (§3): Robust networks exhibit much higher similarity across spatially distant layers compared to non-robust networks,which have far more localized similarity.
The latter thusexhibit the expected specialization in layers, which is unexpectedly missing in the former.
Further, we find reducingthe width of the network has a similar effect to increasing the budget used in robust training, with both changesincreasing long-range similarity among the learned representations.
This indicates that the relative capacity of anetwork drops as the budget is increased.
Early layers are largely unaffected by adversarial examples (§4): The representations of perturbed inputs from non-robust and robust networks are, as expected, starkly different.
On the otherhand, the representations of benign and perturbed inputs from robust networks are indistinguishablefrom one another with regards to representation similarity metrics.
Deeper layers overfit (§5):
For both non-robust and robust training, early layers in the networkconverge within the first few epochs, and exhibit very little variation in subsequent training.
Thisindicates they perform similar roles in both benign and robust networks.
Deeper layers convergeslower while visiting drastically different representations over the course of training.
These differentrepresentations usually occur when the loss indicates overfitting, leading us to conclude that deeperlayers overfit during robust training .
Visually dissimilar adversarial constraints can lead to similar robust representations (§6):
Regardless of the adversarial constraint, we find a high degree of similarity between the representationsof perturbed and benign inputs for robust networks.
Representations from networks robust to different,but allied ( e.g. ℓ p ), threat models are similar across attack strength.
We find surprising similaritiesbetween the representations of visually unrelated threat models at lower attack strengths.
What do our findings imply?
The lack of specialization in robust networks even with heavilyoverparametrized architectures suggests a need for explicit tools to differentiate between layers duringtraining.
Further, the overfitting we find in deeper layers of neural networks is an indication thatlayer-wise specialized regularization methods need to be adopted for robust training.
We hope thelessons from this paper and the accompanying open-source code spur the development of robustlearning-specific architectures and training methods.
Paper Layout.
We provide an overview of RS tools and robust training methods in §2.
We comparerepresentations from non-robust and robust networks using benign inputs in §3.
§4 repeats thisanalysis with perturbed inputs.
The evolution of robust representations over training is studied in §5while the impact of different threat models is in §6.
§7 discusses lessons and limitations.
Background and Setup
We provide a brief overview of the techniques we use to train robust networks and the tools we use toprobe the intermediate representations of neural networks.
Further related work is in Appendix A.
2.1 Adversarial Examples and Robust Learning
Adversarial examples are perturbed inputs to machine learning models that intentionally inducemisclassification [4, 36].
For an input x , an untargeted adversarial example with respect to a model fand loss function ℓ ( · ) is generated by adding a perturbation δ such that δ = arg max x + δ ∈ N ( x ) ℓ ( f ( x +δ ) , y ) , where y is the label of the original class and N ( · ) represents the adversarial constraint orneighborhood.
The most commonly used constraints are L p -ball based, although we also experimentwith other unrelated classes of attacks, such as JPEG, Gabor, or Snow based perturbations [20].
We characterize activations of internal layer in a network as: 1) Benign representation: if input isnon-adversarial 2) Adversarial representations: if input is an adversarial example.
[Table]
Adversarial training [24] has emerged as the most effectivemethods to train robust classifiers [7], where it accounts foradversarial examples during training.
It modifies the empirical risk minimization procedure in non-robust trainingto a min-max optimization, where the inner maximizationfocuses on finding the strongest adversarial example foreach input during training:
[Equation]
where D is a data distribution and f θ is a model with parameters θ .
Newer methods, like TRADES[46], attempt to improve on adversarial training by better balancing the tradeoff between robustnessand accuracy.
We refer to models trained using methods that account for adversarial examples asrobust models and those trained without these methods as non-robust models.
We refer to accuracy on non-perturbed input as benign accuracy and on adversarial examples asrobust accuracy .
Adversarial training is well known to trade-off benign accuracy to gain robustnessand exhibits a high generalization gap in robust accuracy (as we show in Table 1 for CIFAR-10dataset).
2.2 Probing Neural Network Representations
Representation similarity (RS) metrics allow for the quantitative comparison of the activations ofdifferent sets of neurons on a given dataset.
Comparing learned, internal representations within andbetween neural networks of different architectures is important for understanding the performanceand behavior of these networks under changing conditions.
Several metrics have been proposed forthis purpose: Canonical Correlation Analysis (CCA) [17] and its extensions, Singular Vector CCA(SVCCA) [31] and Projection Weighted CCA (PWCCA) [26], Centered Kernel Alignment (CKA)[21], the Orthogonal Procrustes distance [13] and model stitching [2, 8].
These metrics can all handledifferent sized representations, but require that the number of datapoints be the same.
Other workhas used the fitting of sparse linear layers to understand the behavior of deeper layer representations[41, 34].
We choose CKA to measure representation similarity, due to its desirable properties such as orthogonaland isotropic invariance, and wide use in previous work [28, 29, 30, 40].
We also find CKA to becomputationally more efficient ( 10 × and 30 × faster than Procrustes and SVCCA respectively), whileyielding similar high-level insights (see Appendix B.1.).
Ding et al.
[13] have shown that CKA isnevertheless insensitive to deletions of principal components of representations that strongly correlatewith model accuracy.
However, this property does not impact the manner in which we use CKA.
Centred Kernel Alignment (CKA): Let X ∈ R n × p 1 with rows x i , and Y ∈ R n × p 2 with rows y ibe the activation matrices of layers with p 1 and p 2 neurons respectively, defined on the same set of npoints.
These activation matrices are then the representations induced by a particular layer.
We alsodefine a n × n centering matrix H =
I n − 11 ⊤ .
The CKA between these activation matrices is then
[Equation]
where K ij = k
( x i , x j ) ,
L ij = l ( y i , y j ) , with k and l being kernels.
When the kernels are linear andthe activation matrices are centered, we get the Linear CKA between activation matrices to be
[Equation]
Since Kornblith et al.
[21] find the use of a linear kernel to be as accurate as and much faster than the
RBF kernel, we focus only on linear kernels.
To reduce memory usage, we use online CKA whichuses batching to get an unbiased estimate of CKA [29].
2.3 Experimental Setup
We provide a brief overview of our experimental setup here.
All of our experiments were run onmachines with either NVidia Titan RTX GPUs with 24GB of memory or RTX A4000 GPUs with16GB of memory.
Further details are in Appendix A.
Models and datasets.
We consider three commonly used image dataset: CIFAR-10 [22], ImageNette [19], and ImageWoof [19], where latter two are subsets of ImageNet dataset [10], whichwe use due to the high cost of adversarial training on the full ImageNet dataset.
We use the Wide
Resnet [44] class of models on CIFAR-10 dataset and ImageNet-based ResNets for ImageNette and
Imagewoof datasets.
Adversarial training.
We follow standard convention with ℓ ∞ perturbations and use ϵ = 8255 for
CIFAR-10 and ϵ = 4255 for ImageNet based datasets.
We use 10-step projected gradient descent(PGD) attack during training and use 20 steps at test time to calculate adversarial representations.
Representation similarity.
We will be using mainly using CKA to compare activation matricesderived from different benign or adversarial images.
In online CKA, we use a batch-size of 1024and take 3 passes over the dataset to reduce any stochasticity in the output similarity score.
Inlayer-wise similarity analyses
, we consider all layers, including convolutions, pooling, activation, andbatch-normalization layers.
How do representations from robust and non-robust networks differ?
Robust training is expected to be a harder training objective than non-robust training, since it optimizesfor both accuracy and robustness.
The resulting trade-off is also reflected in final performance, wherethe benign accuracy of robust networks is much lower than non-robust networks [24].
In this section,we aim to understand the impact of robust training on internal representations, in particular bycomparing benign representations in non-robust and robust networks.
Further experiments and resultsfrom this section are included in Appendix B.
We take the following two-fold approach:1) Comparing characteristics:
Absence of block structure.
When comparing layerwise similarityin non-robust networks, numerous works have revealed a ‘block structure’, with high similaritybetween the layers comprising a block, and low similarity outside it [21, 29, 32] (Non-robust networkplots in Figure 2).We observe that when moving from benign to robust training, while keepingeverything else constant, the block structure significantly fades .
The much weaker block structure inrobustly trained networks is accompanied by the presence of long-range similarities between layers,with layers having high similarity to those much farther away in the network (Figure 2).
2) Direct comparison: Poor similarity between non-robust and robust network representations.
To further investigate the unique characteristics of robust networks, we conduct a cross-layer comparison of benign representations in non-robust and robust networks (Figure 2d).
We find a very lowdegree of similarity across both network representations, which further shows that robust networkshad indeed learned strikingly different representations than non-robust networks.
We also show that both of these observations generalize across architectures, datasets and threatmodels.
We now delve deeper into analyzing cross-layer characteristics of robust networks representations.
[Figure]
We measure the similarity of benign representations for three datasets across all pairs of layers in both non-robust and robust networks
Robust networks have an amplified degree of crosslayer similarity even at long-range, which diminishes the well known block structure characteristic observed in non-robust networks [29]
We observe low similarity between representations of non-robust and robust models with identical experimental setups
Impact of robust training strength.
We investigate this disappearance of block structure in robustlytrained networks further by varying both the adversarial budget ( ϵ ) used for training and the width ofthe network (Figure 3).
As the training ϵ increases, the block structure becomes fainter, eventuallydisappearing entirely.
The block structure is also impacted by network width, with it being marginallymore pronounced in larger networks.
However, this distinction is largely absent at larger values ofϵ .
This establishes that increasing the value of ϵ is making the task more complex, so maintainingthe same model capacity leads to a decrease in relative capacity (indicated by high similarity acrossrepresentations).
Impact of architecture.
Previously Nguyen et al.
[29] observed that increasing network width, thuscapacity, leads to emergence of block-structure in non-robust networks.
Thus we incrementallyincrease the width of robust Wide-ResNet architecture on CIFAR-10 dataset and measure the crosslayer representation similarity of benign features (Figure 3, expanded on in Appendix B).
Evenafter increasing network capacity by an order of magnitude, we don’t see significant variation inthe characteristics of robust network representations.
These results suggest that increasing width inrobust networks doesn’t lead to a drastic shift in similarity of internal layer representations.
Probing the impact of adversarial perturbations on representations
Robust training aims to learn both high quality benign and robust representations.
In the previoussection we characterized the impact of robust training on benign representations.
In this section,we focus on understanding the characteristics of adversarial representations in robust networks.
Detailed results, as well as ablations on architecture and threat model, are available in Appendix C.
We first analyze how adversarial perturbations distorts the internal representations as we go deeperinto the network, i.e., compare adversarial and benign internal representations.
Next we analyzewhether the distortions introduced by adversarial perturbations can also provide insights into whysuch perturbations transfer between different networks.
4.1 Impact of adversarial perturbations across layers in non-robust and robust networks
In this experiment, we measure the CKA similarity between benign image representations andcorresponding adversarial images representations. [Figure] [Figure]
Early layers are the least affected.
Adversarial perturbations are crafted to corrupt final layerrepresentations.
In our experiments, as expected, the cross-layer similarity score decays to near zeroby the final layer (Figure 4).
However, we observe another surprising trend in which adversarialrepresentations share a high degree of similarity with a fraction (up to one-third for ImageNette and
Imagewoof datasets) of early layers.
We find this observation holds across architectures and budgets,indicating that earlier layers representations, even in benign networks, are robust to adversarialperturbations
Robust networks successfully prevent the distortion of internal representations.
For robustlytrained networks, benign and robust representations maintain a high degree of similarity throughoutthe network, with divergence only occurring in the last 10 layers or so.
In our ablation on network size,we see that increasing the size of the network leads to increased differentiation, implying sufficientcapacity for increased separability between benign and perturbed representations.
4.2 Understanding transferability through the
lens of internal representation similarity
[Figure]
non-robust (NR) Wide-ResNet networks of varying width (e.g., R-5 refers to Robust W RN -28-5). Low robust accuracy implies high transferability.
(b, c, d) We measure RS between source and target networks, where these representations are extracted using adversarial examples generated for the source network. When both networks are trained in the same fashion (benign or robust), the layer wise similarity is generally high at earlier layers but drops off towards the end. In contrast, it drops right after the first layer from robust to non-robust network, a set of networks which also have least transferability.
It is well-known that adversarial examples are transferable from one architecture to another, i.e.,adversarial examples generated on one network achieve the misclassification objective on others too.
In Figure 5a, we measure the transferability between non-robust and robust networks.
Intriguingly, aspreviously validated by [7], transferability is higher among some networks (non-robust to non-robust)while much poorer for others (robust to non-robust).
High transferability indicates that the targetnetwork has learned a similar decision boundary to the source network.
We aim to better understandthis phenomenon by comparing the internal representations of source and target networks.
We firstgenerate adversarial examples, and their corresponding internal layer representations, from a sourcenetwork (W RN -28-10 in this case).
We compare them with internal layer representations of theseadversarial examples on the target network.
We consider transfer from non-robust to robust, robust torobust, and robust to non-robust networks 2 (Figure 5).
For transfer from non-robust to non-robust or robust-robust, where highest transferability occurs,we observe high degree of similarity between representations at early layers.
It indicates that thetransferred adversarial examples distort the internal representations in a similar fashion as sourcenetwork.
The most intriguing aspect in Figure 5a is extremely poor transferability from robustto non-robust networks.
We establish that this is due to significant differences in representationswhere, even across identical architectures, we observe a large drop in CKA right after input layer.
It shows that adversarial perturbations from robust nework distorts the internal representations ofnon-robust network in a strikingly different manner (even at very early layers), which likely rendersthem ineffective to cause an adversarial effect at the output of the network.
Evolution of internal representation dynamics during robust training
In this section, we track the evolution of representations learned by robust training methods overthe course of training epochs in the optimization process.
We study the following two convergenceproperties: 1) Rate of convergence at different layers, and 2) Stability of different layers throughoutlearning.
As in earlier sections, we use CKA to measure similarity, but in this section we comparethe benign representations of the same layer across time .
We present our results for Wide-ResNetarchitecture in Figure 6, where we compare the representations at each epoch to the final epoch(epoch 100).
We further investigate across architectures, layers, and attack budgets in Appendix D. [Figure]
Early layers converge faster than later layers.
Benign representations after the first two blocks of
W RN -28-10 are highly similar to the representation obtained at the end of training, after just tenepochs (Figure 6).
This is true for all the training methods considered.
In Appendix D, we show thislargely holds true across architectures and smaller budgets.
Our observation relates to the intuitionthat early layers learn simpler features [23, 43, 45], thus are learned much faster than complex featurelearned by later layers.
Only with high budget ( ϵ = 16255 ), i.e., stronger attacks, we observe a deviationfrom this trend where second block convergence is slower.
Overfitting is predominantly visible in later layers.
From the cross-epoch heatmap of similaritiesbetween adversarial representations (Appendix D), we observe that later layer representations onlyexhibit similarity off-diagonal at the beginning (until epoch 30) and end of training.
This clearlyshows that multiple local minima are found during training.
This is true to a less pronounced degreefor benign representations.
In Fig.
6, we also observe the adversarial validation loss increasing afteraround 30 epochs, while the training loss keeps decreasing, indicating overfitting.
After this point intraining, the CKA similarity between drops with respect to both the epoch 30 and final representationdrops significantly in later layers, while the earlier layers remain unaffected.
This strongly indicatesthat the later layers have overfit to the training data.
Adversarial training reduces stability of internal representations during training.
While subsequent epoch representations tend to be very similar to each other in non-robust training, therethere can be very large jumps in similarity between one epoch and the next, indicating a less stableconvergence.
This effect further accentuates when training with a higher perturbation budget or whentraining a larger network (Appendix D).
How does the threat model impact representations?
Throughout the paper, we have focused on a single threat model: ℓ ∞ perturbations.
In this section,we extend our analysis of robust representations to understand what exactly is the link between thevisual similarity of threat models and the representations of models trained to be robust to them.
Tothis end, we consider four additional threat models: ℓ 2 , JPEG, Gabor and Snow [20] (visualized in
Appendix A).
We also examine the generalizability of our earlier results across these threat models.
For space reasons, all associated figures from this section are in Appendix E.
Do visually similar attacks lead to similar robust representations?
RS gives us a method bywhich we can compare how different classes of adversarial perturbations affect learned representations.
Using CKA, we plotted the cross-layer similarity of robust WRN-28-10 networks adversarially trained
[Figure]
against five different threat models ( ℓ 2 , ℓ ∞ , JPEG, Gabor, and Snow) (Figure 7).
These comparisonslend credence to conventional knowledge about these threat models.
The layerwise similarity plotbetween l 2 and l ∞ robust models is strikingly similar to CKA plots between robust models trained onthe same threat model.
This mirrors theoretical results on correspondences between ℓ p threat models[37].
We also observe a higher average similarity when comparing an ℓ ∞ model with a JPEG modelthan with Snow or Gabor models.
This makes intuitive sense, given the ℓ ∞ constraint inherent inthe JPEG attack.
When using CKA to compare against the Snow threat model, we observe that thehighest average similarity is achieved with Gabor.
This represents a novel insight into these threatclasses, as correspondence between the two was not previously known.
Alignment of budgets across different attacks.
CKA can also give insight into the varying effects ofchanging perturbation strengths within different threat models.
Expanding on our experiments above(all figures are in Appendix E), we compared representations between different threat models whilevarying perturbation strengths.
Between ℓ ∞ and Gabor models, the layerwise similarity structureis much more sensitive to changes in the strength used in ℓ ∞ -robust training than in Gabor-robusttraining.
For more closely related attacks, like ℓ ∞ and JPEG, we see similar degradations of thestructure when changing the strength of either attack.
Furthermore, the structure is largely preservedwhen the attack strengths are changed in tandem.
These results may be useful for developingtechniques for training models to be jointly-robust against different classes of attack.
Layerwise structure across threat models.
We find that across all threat models except Snow,robust and benign activations exhibit high similarity (Appendix E).
Further, changing budgets acrossthe same threat model, we find that ℓ p threat models have representations that evolve more gradually.
Discussion
Limitations.
The key limitations of our study arise from the fact that all discoveries are predicatedon the effectiveness of the RS metric used at discerning differences between representations.
Further,the nature of our study can only unearth correlations between layer-wise and aggregate propertiessuch as accuracy and loss.
Establishing causal links needs further investigation, perhaps inspired byour findings.
Finally, methods to improve robust training are beyond the scope of this paper.
Theyneed to be derived from a careful analysis of the observations in this paper for actionable changes totraining strategies and architectures.
Future Work.
Our observations highlight that robust networks could be improved by encouraginggreater differentiation between layers during training as well as targeted reduction of capacity indeeper layers.
Our results on different threat models indicate a path forward for networks robust tomultiple attackers as well by allowing for
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 3, 4, 5,
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our paper
is of an abstract nature and is quite removed from direct societal impact.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] Having read the guidelines, given the abstract nature of the paper andthe use of publicly available datasets, we do not foresee any ethical concerns from the
publication of this paper.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] All claims in
the paper are of an empirical nature
(b) Did you include complete proofs of all theoretical results?
[N/A] See above
If you ran experiments...
(a) Did you include
the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code toreproduce all figures and results is included in the supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] The baseline setup is in Section 2 and each section including the
Appendix contains additional details as necessary.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Our results are averaged over 3 runs with the variationbeing too small to appear on our plots.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Section 2 contains compute details
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Citations are included
throughout the paper as appropriate
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
All included assets are publicly available.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] See above
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] See above
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]