E XPLORING PERCEPTUAL STRAIGHTNESS IN LEARNED VISUAL REPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans have been shown to use a ”straightened” encoding to represent the natural visual world as it evolves in time (H´enaff et al. 2019).
In the context of discrete video sequences, ”straightened” means that changes between frames follow a more linear path in representation space at progressively deeper levels of processing.
While deep convolutional networks are often proposed as models of human visual processing, many do not straighten natural videos.
In this paper, we explore the relationship between network architecture, robustness, biologicallyinspired filtering mechanisms, and representational straightness in response to time-varying input; we identify curvature as a useful way of evaluating neural network representations.
We find that (1) adversarial training leads to straighter representations in both CNN and transformer-based architectures but (2) this effect is task-dependent, not generalizing to tasks such as segmentation and frameprediction, where straight representations are not favorable for predictions.
Finally, (3) biologically-inspired elements increase straightness in the early stages of a network, but do not guarantee increased straightness in downstream layers of
CNNs.
Our results suggest that the ability of a model to straighten is a useful and easily computed measure of representational robustness and stability, as well as a marker of similarity between human and machine representations.
I NTRODUCTION
Visual input from the natural world evolves over time, and one can think of that change over time as a trajectory in some representation space.
For humans, this trajectory changes at different levels of processing, from input at the retina to brain regions such as V1 and finally to perception, as
Figure 1 illustrates.
We can ask about the shape of that trajectory and consider that there might be advantages to a straighter, less curved, trajectory.
(H´enaff et al., 2019) demonstrated that trajectories are straighter in human perceptual space than in pixel space.
They suggested that a straighter representation may be useful for visual tasks that require extrapolation, such as predicting the future.
Learning a useful visual representation is one of the major goals of computer vision.
Properties like temporal stability, robustness to transformations, and task flexibility – all of which characterize human vision – are often desirable in computer vision representations.
Yet, many existing computer vision models still fail to capture aspects of human vision, despite achieving high accuracy on visual tasks like recognition (Feather et al., 2019; H´enaff et al., 2019).
In (H´enaff et al., 2019) it was found that, while biologically-inspired V1-like transformations yield straighter representations compared to the input domain, popular computer vision models such as the original ImageNet-trained
AlexNet (Krizhevsky et al., 2017) do not.
In an effort to achieve these favorable human-vision properties for computer vision models, there has been much work dedicated to incorporating various aspects of human vision into deep neural networks.
These include modifying network architectures to mimic that seen in the human visual system Huang & Rao (2011), incorporating filter banks matching the receptive field properties of visual neurons Dapello et al.
(2020), as well as enforcing activation properties similar to that seen in visual cortex with properties such as sparsity Wen et al.
Another promising avenue in this area has been in modifying training with adversarial examples.
In directly targeting areas of vulnerability, adversarially robust networks show increased representational robustness, more closely aligning them with their human counterparts (Engstrom et al., 2019b).
[Figure]
In this paper, we explore what makes some learned visual representations straighter than others.
We evaluate a variety of network architectures, both biologically and non-biologically inspired, for representational straightness across layers; we then ask whether training for adversarial robustness in both CNN and transformer-based architectures may lead to the straighter representations generated by human vision.
Because DNNs learn an early representation that differs from what is known about human vision, we also ask if hard-coding that early representation might lead to a trained network with more straightening downstream.
Overall, we find that most deep learning models do not employ straightened representations; however, with modified training schemes (notably adversarial training), many models are capable of relying on a straightened representations.
This capability to utilize a straightened representation may be useful for learning more robust and stable image and video processing systems.
P REVIOUS W ORK
Deep neural networks have been proposed as models of human visual processing, owing to their ability to predict neural response patterns (Yamins & DiCarlo, 2016; Rajalingham et al., 2015; Kell & McDermott, 2019).
As such, there has been much effort to improve the alignment of deep networks to human vision by incorporating known aspects of the human visual system.
Some of these include: simulating the multi-scale V1 receptive fields of early vision (Dapello et al., 2020), adding foveation using a texture-like representation in the periphery at a CNN’s input stage (Deza & Konkle, 2020), and incorporating activation properties of visual neurons such as sparsity Olshausen &
Field (1997); Wen et al. (2016).
Predictive coding, often attributed to biological networks Huang & Rao (2011), has been incorporated into deep networks trained to perform tasks such as video frame prediction Lotter et al.
(2016), using layers that propagate error-signals.
The desire to evaluate the effectiveness of these techniques at creating models of the human visual system, motivated the creation of measures like BrainScore Schrimpf et al.
(2020) that compare models to humans.
One important area in understanding how humans and DNNs differ lies in their response to adversarial examples (Elsayed et al., 2018; Ilyas et al., 2019;
Feather et al., 2022; Dapello et al., 2021).
Adversarial examples, which modify images with changes that are imperceptible to humans, can cause a network to misclassify an image (Goodfellow et al., 2014; Szegedy et al., 2013).
Adversarial training (Madry et al., 2017) improves the misclassfication problem and has been suggested to help networks learn visual representations that are more perceptually aligned with humans Engstrom et al.
(2019b)
; Ilyas et al.
If adversarial training leads to models that are more aligned with human perception, perhaps they also learn straight representations like humans do.
Adversarial training schemes, however, are not biologically plausible, and recent work has identified mechanisms that are better supported by vision science (Dapello et al., 2020;
Guo et al., 2022; Dapello et al., 2021).
In the context of representation learning in computer vision, adversarially robust models have also been shown to do better at transfer learning than their non-adversarially robust counterparts (Davchev et al., 2019), and adversarially robust features can be used directly for tasks like image generation and in-painting (Santurkar et al., 2019).
Adversarial training schemes have also been developed for tasks other than classification like semantic segmentation (Xu et al., 2021).
In this paper, we build on work around adversarial robustness by evaluating if increasing this robustness leads to straightened representations like those found in human spatiotemporal processing.
There have been various efforts to incorporate constraints that reduce curvature and increase straightness in temporal representation learning.
Slow feature analysis (Berkes & Wiskott, 2005) encourages a learned representation to change slowly over time, but does not encourage straightness explicitly.
Another early attempt to add a straightness constraint to representation learning (Goroshin et al., 2015) focused on video frame prediction for unsupervised learning.
We follow in the same vein of this work, evaluating straightness as a metric for encouraging a robust, stable, representation when learning to solve tasks.
M ETHODS 3. M EASURING S TRAIGHTNESS AS L OW C URVATURE
Representational straightness can be evaluated as a reduction in curvature.
For a temporal sequence, such as a video, curvature is defined as the angle between the vectors representing the difference between consecutive frames.
Let x refer to a representation of a video of length T , with x t being a representation of one frame of a video at time step t .
The representation may be at any stage of the processing pipeline, from a vector of raw input pixels from the video frame, to the activations of a network’s hidden layer.
Then, v t represents the difference between successive frames:
[Equation]
We can find the curvature at time t by finding the angle between successive ˆ v t , which we call c t :
[Equation]
The global curvature of a video sequence is then simply the mean angle over all time steps.
Note there are T − 1 time steps due to curvature being evaluated on frame differences:
[Equation]
This is the formulation proposed by (H ´ enaff et al., 2019).
One can compute this global curvature for any representation of a video sequence over time, either on the vector of pixels (likely not very straight), or one can apply it to a representation of that video, e.g. at any layer of a neural network model.
In this paper, we often report change in curvature with respect to the pixel-value input to show if a model’s representation is more straight at later stages of processing.
Change in curvature is the representational curvature minus the pixel curvature.
Note that curvature is distinct from simple cosine similarity in that curvature is calculated on frame differences ( v t ), whereas cosine similarity depends on the angle between the frame vectors themselves ( x t ).
Curvature can be thought of as a first-order variant of cosine similarity (see Figureand Section A.1).
For a complete description of how curvature was determined for each model, see
Section A.2.2.
C URVATURE AND R OBUSTNESS IN C ONVOLUTIONAL O BJECT R ECOGNITION M ODELS
We tested a variety of models for output curvature, to investigate the relationship between model type and curvature of the output layer.
As shown in Figure 3, we found non-adversarially trained image recognition models to have the highest output curvature.
All adversarially trained models
[Figure]
The datasets are IN:ImageNet (Deng et al., 2009), RIN:RestrictedImageNet (Santurkar et al., 2019),CF:CIFAR10 (Krizhevsky et al., 2014),
VOC:PASCAL VOC 2012 (Everingham et al., 2010), CIT:Cityscapes (Cordts et al., 2016),
KITTI:KITTI (Geiger et al., 2012).
Exact model details can be found in the supplemental data.
The y axis shows curvature of each model at the output representation layer relative to the curvature of the input video.
For image classification models the output representation is the penultimate layer (just before classification).
This is sometimes referred to as the output layer.
have lower curvature than their non-adversarially trained counterparts, as well as overall, with the majority reducing output curvature below that of the input pixels.
Self-supervised DINO (Caron et al., 2021) models have similar output curvature values to their supervised counterparts – despite
DINO models having been shown to have more semantically meaningful feature correspondences.
4. A DVERSARIAL A TTACK T YPE , S TRENGTH , AND C URVATURE
Given the increased straightness seen in adversarially robust image recognition / object detection models, we investigated the relationship between the type and strength of adversarial attack and the resulting curvature of the model’s output.
To evaluate the effect of these attacks on curvature, we compare a set of ResNet50 networks (He et al., 2016), trained on CIFAR-10 (Krizhevsky et al., 2014),
ImageNet (Deng et al., 2009), and Restricted ImageNet (a subset of ImageNet (Engstrom et al., 2019b; Ilyas et al., 2019)), with and without adversarial training and measure the output curvature (Fig 5).
The adversarially trained models are trained using projected gradient descent with l 2 or l ∞ norms at different perturbation levels (Madry et al., 2017; Engstrom et al., 2019a).
We use ResNet50 adversarially trained with projected gradient descent because that model type has been shown to be more aligned with human perception (Engstrom et al., 2019b; Feather et al., 2022;
Harrington & Deza, 2022).
During projected gradient descent, adversarial examples are created to train the model to correctly classify them.
The adversarial example is created by adding noise to an image.
The amount of noise is bounded by a norm (usually l 2 or l ∞ set to a level ϵ ).
We include some models trained on CIFAR-10 to understand how perturbation level ϵ interacts with straightness (CIFAR-10 models are faster to train, so more models are available).
reduce curvature in the output layer of ResNet.
Increased attack strength ( ϵ ) decreases curvature, with l 2 attacks most reducing
Data We find that, as predicted, output curvature is highest for non-adversarially trained networks.
l ∞ attacks decrease output curvature, with larger values of ϵ leading to decreased curvature.
l 2 -attacked networks have the straightest output curvatures, however strength of attack does not greatly affect the output curvature.
The l 2 norm models may achieve straighter representations than the l ∞ models because they allow for greater ϵ in training.
Among l 2 models, higher ϵ values may need to be tested to see a difference in curvature among l 2 norm trained models.
Overall, we show that adversarial robustness to larger perturbations leads to straighter representations.
Note that while we show that curvature is predictive of adversarial robustness, this does not translate to improved accuracy on the test set (see Section A.4).
We argue however, that while curvature is not predictive of test accuracy directly, adversarial robustness is associated with many other model benefits, including representational stability and alignment with human perception.
We also explore how curvature relates to other types of robustness by evaluating leaders on the ImageNet-c benchmark (Hendrycks & Dietterich, 2019) and and antialiasing CNNs (Zhang, 2019).
We find that none of these models straightened natural videos like the adversarially robust models (See Fig 13 for models robust to common corruptions: MAE, MAE-DAT, YOLO-d5-HAT; See Fig 15 for models robust to shifts: anti-aliasing ResNet50, VGG, AlexNet).
Even though models trained for other types of robustness behave more like humans, the curvature measure reveals that those types of robustness are not sufficient to create the temporal representational stability observed in humans (H´enaff et al.,
Adversarial robustness may stand-out as producing straightness because it corresponds to more human-meaningful representations as illustrated through feature inversion (Engstrom et al., 2019b).
4. C URVATURE AND R EPRESENTATIONAL S TABILITY
One potential benefit of a model with lower curvature is a more stable representation over time. In other words, lower curvature may be a reflection of increased stability of predictions for videos. To test this, we further analyzed an adversarially robust and non-robust ResNet50 model on the test set videos from the VGGSound dataset (Chen et al., 2020).
We measured both the curvature in the final layer, as well as the standard deviation of the predicted probability of the ground truth labels (Fig 6).
The adversarially robust ResNet50 has an overall lower curvature than the standard-trained model over the VGGSound videos (standard mean: 111 . 427886 , adversarially-robust: 104 . 158480 ), and furthermore it has an overall lower standard deviation (std) in predicted probability for the true labels (standard std: 0 .
150474 , adversarially-robust std: 0 . 105826 ).
This indicates that adversarially robust models, which have lower curvature, also show more stable predictions, despite these videos being out of distribution from the training set.
Thus curvature could used as a computationally cheap proxy for the temporal stability of network representations and predictions.
C URVATURE IN V ISUAL T RANSFORMER M ODELS FOR O BJECT R ECOGNITION
Vision transformers (ViT) have very different architectures than CNNs.
They replace convolutional elements with self-attention on visual tokens and achieve state of the art performance on a variety of visual tasks.
In addition, these networks are more adversarially robust than standard-trained CNNs, which is attributed to their higher-level and more generalizable features (Shao et al., 2021).
Despite this, we find that standard-trained ViT (ViT Base, patch 16 ) does not reduce curvature on the natural video sequences tested (Fig 3).
Self-supervised training of ViT (base, patch 16 ) using the DINO method (Caron et al., 2021) also results in a more curved representation than the input space.
However, this DINO trained ViT has a more straight representation than the baseline ViT, despite the two models sharing the same architecture.
This suggests that self-supervised training may be better than supervised training for obtaining straight representations.
No transformer model except for CrossViT daggar (Chen et al., 2021) reduced curvature in any layer compared to the pixel input (Fig 7).
The structure of CrossViT models is unique in that they split into multiple paths, and the reduction in curvature we found interestingly only happens in one pathway of the model and does not persist after the two paths recombine.
A possible explanation for this is that the CrossViT is able to straighten in the part of the model that splits because that pathway utilizes a multi-scale representation, a known aspect of human vision representations such as V1 which we show in Sec. 7 to induce straightening.
C URVATURE AND R OBUSTNESS IN I MAGE S EGMENTATION AND F RAME P REDICTION M ODELS
Output straightness over time should be a favorable, stabilizing property for object detection models, where objects often stay continuously in-frame over many frames of a video.
For image segmentation models on the other hand, predictions lie close to low-level pixel space and change on a frame-by-frame basis.
Therefore, for the task of predicting the dynamically changing segmentation map of an input movie, straightness of the output representation would not be desirable (assuming the input movie contains motion).
To test this prediction, we measured output curvature for a family of 12 image segmentation models from (Xu et al., 2021), consisting of two different architectures, trained on two different datasets, with a non-adversarial as well as two different adversarial training schemes.
Unlike object detection models, where adversarial training greatly reduces output curvature, adversarial training in segmentation models only slightly reduces curvature compared to non-robust segmentation models (Fig 3), with all adversarially trained segmentation models displaying output curvature higher than the input pixel curvature.
Investigating curvature change over model layers (Fig 8), reveals that this trend is consistent throughout all layers of the model.
This highlights the importance of task on straightness, and supports the idea that output curvature reduction is not sim- ply the result of any adversarial training scheme – only when temporal stability is beneficial does adversarial training promote straightness.
Another DNN task for which output predictions are low level, is next-frame prediction for video sequences.
As in image segmentation models, good predictions are expected to vary dynamically frame-to-frame, and representational straightness at and near the output layer is not a favorable quality in response to videos containing motion.
Rather, for a frame-prediction network, a favorable quality would be equal curvature at the input and output.
This is because for a well-performing model, the output prediction over many frames is the same as the input video; two identical videos have identical curvature.
To evaluate this, we test the layer-evolution of curvature for three variants of PredNet (Lotter et al., 2016), a network inspired by predictive coding, which is trained to predict the next frame in a video sequence (Fig 16 bottom).
Indeed, for all pretrained variants of PredNet tested, the output layer maintains a very similar curvature value to that of its input frame, with final output curvature closer than all other models to that of the input (Fig 3).
For all pretrained variants, the model curvatures increase in the representation of the first model block, then re-straighten the representation throughout the rest of the network before returning to the original pixel curvature.
We also investigated a variant of PredNet which had been fine-tuned to predict multiple frames in the future (PredNet Extrap KITTI), which we find only slightly reduces model curvature, with a similar trajectory over layers.
Since PredNet is trained on videos, we were able to directly see the impact of out-of-distribution compared to in-distribution data on straightness (Fig 16).
We found that PredNet models straightened more on in-distribution data, but the effect was small and did not change the overall trend we report using the stimuli from (H ´ enaff et al., 2019)).
For the variant of PredNet in which predictive error for all layers is incorporated into the loss during training (PredNet Lall KITTI), this increase is much less pronounced, pointing to a more temporally-stable representation for this network globally.
This is in agreement with previous findings, showing that while this globally-trained PredNet slightly under-performs at the original frameprediction task, when using its network backbone to perform a classification task, this least-curved
PredNet model is the best performer.
In other words, the network with the most straight representation performs best in the context of a different task, where temporal stability is favorable.
We also tested the utility of straightness as a training signal, adding a curvature constraint to the first (R3) and second (R2) layers of PredNet during training to create ”straight” PredNet models.
We evaluate how the constraint affects the model’s internal representation and the final frame prediction.
For both models, curvature is greatly reduced in R2 and R3 compared to the input curvature (Fig 16 bottom).
Because of PredNet’s top-down and bottom-up feedback structure, the constraint on layer R3 does not affect any other layers of the model.
Although we find that it is possible to train
PredNet to straighten and still get next-frame predictions that has lower mean square error (MSE) than copying the previous frame, we are not able to achieve equivalent or better prediction MSE with the straight models compared to the original on KITTI (Geiger et al., 2012) (original L0 PredNet: 0 .[-
-]
MSE, straight R2 PredNet: 0 .
MSE, compared to previous frame: 0 . {+
+}
MSE).
See Fig 17 for example predictions.
While these straightened PredNet variants do not achieve state of the art, we argue that these models may display other advantages such as robustness and similarity to human representations.
C URVATURE IN B IOLOGICALLY I NSPIRED M ODELS
We investigated straightness for a variety of both parametric (learned) and non-parametric biologically-inspired models.
Given straightness is thought to increase over progressively deeper layers of visual processing, we align these networks along the visual processing areas they are most closely matched to (Fig 9).
Some models match to visual areas explicitly in their architecture like the non-parametric Henaffbio (H´enaff et al., 2021), which is a two-stage model based on centersurround filters follows by oriented Gabor filters, and Brucenet (Brown et al., 2021), which is based on summary statistics of a steerable pyramid.
This explicit match to biological vision also holds for the early layers VOneNetCornets
(Dapello et al., 2020), which combines a V1 filter front-end and
Cornets as a backbone (Cornets being a shallow neural network that has layers matched to regions of the human brain (Kubilius et al., 2019)).
For the adversarially trained Visual Transformer network CrossViTRotAdv (Berrios & Deza, 2022), these layers are those best matched by BrainScore (a benchmark for how similar a model is to the human brain at object recognition Schrimpf et al.
(2020)), for V4 and IT layers (Schrimpf et al., 2020).
For all biologically-inspired models except for VOneNetCornets, curvature progressively decreases through deeper network layers.
For VOneNetCornets, curvature decreases up until the V1 layer in which a noise term capturing neural stochasticity is added; curvature then strongly increases, far above the pixel-curvature baseline.
To determine if this increase in curvature was due to the added noise, we tested the same model with the noise term set to zero.
While this reduced the downstream curvature after the V1 layer, this change did not eliminate the curvature increase present
The increase in curvature for VOneNet at later layers suggest that making the front-end of a deep network more like biologically-inspired models does not suffice to get a straightened representation downstream in a deep network.
This is interesting because VOneNet is is reported to be more adversarially robust to white-box attacks than a standard trained CNN.
This suggests that adversarial training , not the property of adversarial robustness itself, leads to straightened representations in deep neural networks.
Our finding supports (Dapello et al., 2021) who found that the neural population geometry of adversarial trained models was noticeably different from VOneNet and other models trained with neural stochasticity mechanisms.
However, it is puzzling that adversarial training, which is biologically implausible, would lead to straighter representations than biologically inspired mechanisms.
More constraints or modifications may be needed to achieve straight representations with biologically plausible methods.
D ISCUSSION
We show that one can use the change in model representational curvature as a simple and computationally cheap metric for evaluating both image and video models across a variety of tasks.
For a variety of image task models, we show that training with strong white-box adversarial attacks reduces output curvature.
However, this is only true for tasks like image classification for which temporal stability is favorable, and not for semantic segmentation and video frame prediction.
Moreover, we show that self-supervised training does not produce straightened representations.
In evaluating both
CNN and transformer-base architectures we show that curvature reduction is mainly a product of task and training procedure, not deep neural network architecture alone.
In evaluating curvature over layers in biologically inspired models, we show that biologically inspired mechanisms work to reduce curvature in a model’s representation, even more so than adversarial training.
However, the simple addition of non-parametric biologically inspired filtering mechanisms at the input of a model are insufficient to maintain output curvature.
In summary, we find that models with more known aspects of human vision like robustness have a greater ability to straighten natural videos.
Our results suggests that the ability to straighten might be a useful and easily computed measure of how similar visual representations are to humans.
These results identify representational curvature as a common thread between biologically inspired and adversarially robust models, and highlight the benefits and limitations of these techniques in creating temporally-stable representations.
R EPRODUCIBILITY
To reproduce our results, links to all the models analyzed can be found in the supplemental material in the Network Comparison Spreadsheet.
Sources for the stimuli are in Sec A.2.2 and in the code base linked for the HenaffBio model in Network Comparison Spreadsheet.
Straightness can be determined by calculating curvature using Sec 3 and A.1.
The models, stimuli, and curvature function together should be sufficient to reproduce our results.
A A PPENDIX A. C URVATURE VS C OSINE S IMILARITY
cosine similarity (vectors) :
= cos ( θ ) =
A · B ∥
A ∥∥ B ∥ (5) cosine similarity (video frames) :
= cos ( θ ) = x t · x t + ∥ x t ∥∥ x t +1 ∥ (6) curvature :=
c t = arccos (ˆ v t · ˆ v t +1 ) (7) cosine curvature :
=
ˆ v t · ˆ v t +1 = v t · v t + ∥ v t ∥∥ v t +1 ∥
= cos ( c t ) (8)
A. M ODELS A.2. M ODEL S OURCES
All deep neural networks we analyzed were pretrained.
Exact links to code and weights can be found in the supplemental material in the Network Comparison Spreadsheet.
The standard ImageNettrained ResNet (He et al., 2016) model was downloaded from PyTorch’s model zoo (Paszke et al., 2019).
Adversarially robust ResNet models were all downloaded from (Engstrom et al., 2019a).
The adversarially robust ResNets were trained using projected gradient descent.
All ViT (Dosovitskiy et al., 2020) and standard trained CrossViT daggar (Chen et al., 2021) models were downloaded from the timm library (Wightman, 2019).
All CrossViT daggar adversarially robust and rotationally invariant models were downloaded from the repository of (Berrios & Deza, 2022).
The adversarially robust CrossViTs were trained with fast gradient sign method as stated in (Berrios & Deza, 2022).
DINO models were downloaded from the DINO repository (Caron et al., 2021), while PredNet models were downloaded from (Lotter et al., 2016).
A.2. M ODEL A NALYSIS P ROCEDURE
We showed each model the same 12 natural videos that were used in the psychophysics experiments of (H ´ enaff et al., 2019).
The videos were taken from the Chicago Motion Database (at University of
Chicago, 2022), the film ‘Dogville’, Lions Gate Entertainment (2003), and LIVE Video Quality
Database (Seshadrinathan et al., 2010b;a).
The videos were grayscale, consisting of 11 frames each of 512 × 512 pixels, capturing natural motion such as rippling ocean water or a person walking through a crowded street.
We resized the video frames to be 224 × 224 for all deep networks and 256 × 256 for bio-models that use steerable pyramids.
One limitation of this work is that we did not evaluate models on larger video datasets, but we wanted to use psychophysically validated stimuli for our analyses.
For each model, we recorded its activations at intermediate and final layers for each video.
We then found the global curvature for each stage of the model using equation 4 where we used the flattened model activations as the input x t to the curvature procedure.
We compared the global curvature at each layer of the model to the curvature of the video in pixel space.
Models that straighten are defined as models that have a lower global curvature at deeper layers.
When comparing the curvature of different model layers, we chose not to reduce the dimensionality of each layer activation to be the same across stages.
Although principle components analysis (PCA) was sometimes used in (H´enaff et al., 2019) when expressing curvature, they did not use it in their analysis of deep networks.
Furthermore, while an architecture’s inherent dimensionality is likely relevant to a representation’s curvature, we preferred not to introduce any additional transformations that would influence the measured curvature.
We also found that performing PCA did not greatly affect trends like adversarial training leading to more straight representations See Figure 10 for examples of curvature for different numbers of principle components on two ResNet models.
A.2. C OMPUTE
Our methods do not require large compute.
All individual model analyses can be run on CPU.
We used a single GPU to speed up getting the features activations at each layer to the order of minutes per model.
[Figure]
[Figure]
Model families show very similar curvature to that of their output representation layers 3.
The final stage of a network for image classification models is the last fully connected layer.
We see similar trends in curvature across model families as in Fig 3.
We chose to report the the curvature on the layer just before classification because this is the layer that would be most commonly used as the representational backbone for other tasks.
A. A DVERSARIAL A CCURACY AND C URVATURE
[Figure]
[Figure]
A. N EGATIVE S OCIETAL I MPACTS
We believe there are few negative societal impacts of this paper.
Our work was exploratory and did not introduce any new models.
However, we note that the development of machine vision systems, which can operate at or above the level of performance of humans at certain tasks, may lead to negative societal impacts such as the loss of jobs and industries as human workers are replaced.