Original: B RAIN - LIKE REPRESENTATIONAL STRAIGHTENING OF NATURAL MOVIES IN ROBUST FEEDFORWARD NEURAL NETWORKS
Revised: B RAIN - LIKE REPRESENTATIONAL STRAIGHTENING OF NATURAL MOVIES IN ROBUST FEEDFORWARD NEURAL NETWORKS

Original: A BSTRACT
Revised: A BSTRACT

Original: Representational straightening refers to a decrease in curvature of visual feature representations of a sequence of frames taken from natural movies.
Revised: Representational straightening refers to a decrease in curvature of visual feature representations of a sequence of frames taken from natural movies.

Original: Prior work established straightening in neural representations of the primate primary visual cortex (V1) and perceptual straightening in human behavior as a hallmark of biological vision in contrast to artificial feedforward neural networks which did not demonstrate this phenomenon as they were not explicitly optimized to produce temporally predictable movie representations.
Revised: Prior work established straightening in neural representations of the primate primary visual cortex (V1) and perceptual straightening in human behavior as a hallmark of biological vision in contrast to artificial feedforward neural networks which did not demonstrate this phenomenon as they were not explicitly optimized to produce temporally predictable movie representations.

Original: Here, we show robustness to noise in the input image can produce representational straightening in feedforward neural networks.
Revised: Here, we show robustness to noise in the input image can produce representational straightening in feedforward neural networks.

Original: Both adversarial training (AT) and base classifiers for Random
Revised: Both adversarial training (AT) and base classifiers for Random

Original: Smoothing (RS) induced remarkably straightened feature codes.
Revised: Smoothing (RS) induced remarkably straightened feature codes.

Original: Demonstrating their utility within the domain of natural movies, these codes could be inverted to generate intervening movie frames by linear interpolation in the feature space even though they were not trained on these trajectories.
Revised: Demonstrating their utility within the domain of natural movies, these codes could be inverted to generate intervening movie frames by linear interpolation in the feature space even though they were not trained on these trajectories.

Original: Demonstrating their biological utility, we found that AT and RS training improved predictions of neural data in primate V1 over baseline models providing a parsimonious, bio-plausible mechanism – noise in the sensory input stages – for generating representations in early visual cortex.
Revised: Demonstrating their biological utility, we found that AT and RS training improved predictions of neural data in primate V1 over baseline models providing a parsimonious, bio-plausible mechanism – noise in the sensory input stages – for generating representations in early visual cortex.

Original: Finally, we compared the geometric properties of frame representations in these networks to better understand how they produced representations that mimicked the straightening phenomenon from biology.
Revised: Finally, we compared the geometric properties of frame representations in these networks to better understand how they produced representations that mimicked the straightening phenomenon from biology.

Original: Overall, this work elucidating emergent properties of robust neural networks demonstrates that it is not necessary to utilize predictive objectives or train directly on natural movie statistics to achieve models supporting straightened movie representations similar to human perception that also predict V1 neural responses.
Revised: Overall, this work elucidating emergent properties of robust neural networks demonstrates that it is not necessary to utilize predictive objectives or train directly on natural movie statistics to achieve models supporting straightened movie representations similar to human perception that also predict V1 neural responses.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In understanding the principles underlying biological vision, a longstanding debate in computational neuroscience is whether the brain is wired to predict the incoming sensory stimulus, most notably formalized in predictive coding (Rao & Ballard, 1999; Friston, 2009; Millidge et al., 2021), or whether neural circuitry is wired to recognize or discriminate among patterns formed on the sensory epithelium, popularly exemplified by discriminatively trained feedforward neural networks (DiCarlo et al., 2012;
Revised: In understanding the principles underlying biological vision, a longstanding debate in computational neuroscience is whether the brain is wired to predict the incoming sensory stimulus, most notably formalized in predictive coding (Rao & Ballard, 1999; Friston, 2009; Millidge et al., 2021) or whether neural circuitry is wired to recognize or discriminate among patterns formed on the sensory epithelium, popularly exemplified by discriminatively trained feedforward neural networks (DiCarlo et al., 2012;

Original: Tacchetti et al., 2018; Kubilius et al., 2018).
Revised: Tacchetti et al., 2018; Kubilius et al., 2018).

Original: Arguing for a role of prediction in vision, recent work found perceptual straightening of natural movie sequences in human visual perception (H´enaff et al., 2019).
Revised: Arguing for a role of prediction in vision, recent work found perceptual straightening of natural movie sequences in human visual perception (H´enaff et al., 2019).

Original: Such straightening is diagnostic of system whose representation could be linearly read out to perform prediction over time, and the idea of representational straightening resonates with machine learning efforts to create new types of models that achieve equivariant, linear codes for natural movie sequences.
Revised: Such straightening is diagnostic of system whose representation could be linearly read out to perform prediction over time, and the idea of representational straightening resonates with machine learning efforts to create new types of models that achieve equivariant, linear codes for natural movie sequences.

Original: Discriminatively trained networks, however, lack any prediction over time in their supervision.
Revised: Discriminatively trained networks, however, lack any prediction over time in their supervision.

Original: It may not be surprising then that large-scale ANNs trained for classification produce representations that have almost no improvement in straightening relative to the input pixel space, while human observers clearly demonstrated perceptual straightening of natural movie sequences (subsequently also found in neurons of primary visual cortex, V1 (H´enaff et al.,2019; 2021)). This deficiencyin standard feedforward ANNs might suggest a need for new models trained on predictive loss functions ratherthan pure classificationto emulatebiological vision.
Revised: It may not be surprising then that large-scale ANNs trained for classification produce representations that have almost no improvement in straightening relative to the input pixel space, while human observers clearly demonstrated perceptual straightening of natural movie sequences (subsequently also found in neurons of primary visual cortex, V1 (H´enaff et al., deficiency loss functions rather classification emulate vision.

Original: Here, we provide evidence for an alternative viewpoint, that biologically plausible straightening can be achieved in ANNs trained for robust discrimination, without resorting to a prediction objective or natural movies in training.
Revised: Here, we provide evidence for an alternative viewpoint, that biologically plausible straightening can be achieved in ANNs trained for robust discrimination, without resorting to a prediction objective or natural movies in training.

Original: Drawing on insights from emergent properties of adversarially-trained neural networks in producing linearly invertible latent representations, we highlight the link between perceptual straightening of natural movies to invertible latent representations learned from static images (Figure 1).
Revised: Drawing on insights from emergent properties of adversarially-trained neural networks in producing linearly invertible latent representations, we highlight the link between perceptual straightening of natural movies to invertible latent representations learned from static images (Figure 1).

Original: We examine straightening in these robust feedforward ANNs finding that their properties relate to those in the biological vision framework.
Revised: We examine straightening in these robust feedforward ANNs finding that their properties relate to those in the biological vision framework.

Original: The contributions of this work are as follows:
Revised: The contributions of this work are as follows:

Original: We show that robust neural networks give rise to straightened feature representations for natural movies in their feature space, comparable to the straightening measured in the primate brain and human behavior, and completely absent from standard feedforward networks.
Revised: We show that robust neural networks give rise to straightened feature representations for natural movies in their feature space, comparable to the straightening measured in the primate brain and human behavior, and completely absent from standard feedforward networks.

Original: We show that linearly interpolating between the start and end frames of a movie in the output feature space of robust ANNs produces synthetic frames similar to those of the original natural movie sequence in image space.
Revised: We show that linearly interpolating between the start and end frames of a movie in the output feature space of robust ANNs produces synthetic frames similar to those of the original natural movie sequence in image space.

Original: Such invertible linear interpolation is precisely the definition of a temporally predictive feature representation.
Revised: Such invertible linear interpolation is precisely the definition of a temporally predictive feature representation.

Original: Compared to prior models of early visual cortex, robustness to input noise (corruption or adversarial robustness) is significantly better at explaining neural variance measured from V1 neurons than non-robustly trained baseline models, suggesting a new hitherto unconsidered mechanism for learning the representations in early cortical areas that achieves natural movie straightening.
Revised: Compared to prior models of early visual cortex, robustness to input noise (corruption or adversarial robustness) is significantly better at explaining neural variance measured from V1 neurons than non-robustly trained baseline models, suggesting a new hitherto unconsidered mechanism for learning the representations in early cortical areas that achieves natural movie straightening.

Original:  R ELATED WORK 2. M ECHANISMS FOR PRODUCING BRAIN - LIKE REPRESENTATIONS
Revised:  R ELATED WORK 2. M ECHANISMS FOR PRODUCING BRAIN - LIKE REPRESENTATIONS

Original: Feedforward ANNs as models of biological vision.
Revised: Feedforward ANNs as models of biological vision.

Original: Standard feedforward ANNs, although lacking a number of bio-plausible features such as feedback connections or a local learning rule (Whittington & Bogacz, 2019), still can explain the neural variance (Schrimpf et al., 2018) recorded from rodent (Bakhtiari et al., 2021), monkey (Yamins et al., 2014; Bashivan et al., 2019), and human visual cortex (Khaligh-Razavi & Kriegeskorte, 2014; Cichy et al., 2016) better than alternatives which are considered more bio-plausible by using a prediction objective function (e.g., PredNet and
Revised: Standard feedforward ANNs, although lacking a number of bio-plausible features such as feedback connections or a local learning rule (Whittington & Bogacz, 2019), still can explain the neural variance (Schrimpf et al., 2018) recorded from rodent (Bakhtiari et al., 2021), monkey (Yamins et al., 2014; Bashivan et al., 2019), and human visual cortex (Khaligh-Razavi & Kriegeskorte, 2014; Cichy et al., 2016) better than alternatives which are considered more bio-plausible by using a prediction objective function (e.g., PredNet and

Original: CPC (Zhuang et al., 2021; Schrimpf et al., 2020)).
Revised: CPC (Zhuang et al., 2021; Schrimpf et al., 2020)).

Original: Thus, to learn the representations in the brain, regardless of bio-plausibility of mechanisms, feedforward ANNs provide a parsimonious more tractable class of leading models for object recognition in the visual cortex.
Revised: Thus, to learn the representations in the brain, regardless of bio-plausibility of mechanisms, feedforward ANNs provide a parsimonious more tractable class of leading models for object recognition in the visual cortex.

Original:  of primary visual cortex.
Revised: Models of primary visual cortex.

Original: In neuroscience, rather than rely solely a on top-down training objective like standard ANNs do, there has been a tradition of explaining early visual representations using more fundamental principles such as sparse coding and predictive coding as well as invoking unsupervised training (Olshausen & Field, 1996;
Revised: In neuroscience, rather than rely solely a on top-down training objective like standard ANNs do, there has been a tradition of explaining early visual representations using more fundamental principles such as sparse coding and predictive coding as well as invoking unsupervised training (Olshausen & Field, 1996;

Original: Rao & Ballard, 1999).
Revised: Rao & Ballard, 1999).

Original: For example, unsupervised slow feature analysis extracts the slow-varying features from fast-varying signals in movies based on the intuition that most external salient events (such as objects) are persistent in time, and this idea can be used to explain the emergence of complex cells in V (Berkes & Wiskott, 2005).
Revised: For example, unsupervised slow feature analysis extracts the slow-varying features from fast-varying signals in movies based on the intuition that most external salient events (such as objects) are persistent in time, and this idea can be used to explain the emergence of complex cells in V (Berkes & Wiskott, 2005).

Original: Recent work in machine learning has attempted to blend more bottom-up principles with top-down training by experimenting with swapping out ANN early layers with
Revised: Recent work in machine learning has attempted to blend more bottom-up principles with top-down training by experimenting with swapping out ANN early layers with

Original: V1-like models whose filters are inspired from neuroscience studies (Dapello et al., 2020).
Revised: V1-like models whose filters are inspired from neuroscience studies (Dapello et al., 2020).

Original: This blended model turns out to have benefits for classification robustness in the outputs.
Revised: This blended model turns out to have benefits for classification robustness in the outputs.

Original: However, it remains unclear whether there is a form of top-down training that can produce V1-like models.
Revised: However, it remains unclear whether there is a form of top-down training that can produce V1-like models.

Original: Such a mechanism would provide a fundamentally different alternative to prior proposals of creating a
Revised: Such a mechanism would provide a fundamentally different alternative to prior proposals of creating a

Original: V1 through sparse coding or future prediction (H´enaff et al., 2019; 2021).
Revised: V1 through sparse coding or future prediction (H ´ enaff et al., 2019; 2021).

Original: 2. T EMPORAL PREDICTION AND INVERTIBILITY IN NEURAL NETWORKS
Revised: 2. T EMPORAL PREDICTION AND INVERTIBILITY IN NEURAL NETWORKS

Original: Learning to predict over time.
Revised: Learning to predict over time.

Original: Changes in architecture, training diet (movies), and objective (predicting future frames) have all been explored as mechanisms to produce more explicit equivariant representations of natural movies (Lotter et al., 2016; van den Oord et al., 2018).
Revised: Changes in architecture, training diet (movies), and objective (predicting future frames) have all been explored as mechanisms to produce more explicit equivariant representations of natural movies (Lotter et al., 2016; van den Oord et al., 2018).

Original: Directly related to the idea of straightening, penalizing the curvature of representations of frames was used in Learning to linearize (Goroshin et al., 2015) to learn straightened representations from unlabeled videos.
Revised: Directly related to the idea of straightening, penalizing the curvature of representations of frames was used in Learning to linearize (Goroshin et al., 2015) to learn straightened representations from unlabeled videos.

Original: This class of models does not need supervision which makes them more bio-plausible in nature; however, as mentioned in the previous section, they lag behind supervised feedforward
Revised: This class of models does not need supervision which makes them more bio-plausible in nature; however, as mentioned in the previous section, they lag behind supervised feedforward

Original: ANNs both in terms of learning effective representations for object recognition and in producing feature representations that predict neural data.
Revised: ANNs both in terms of learning effective representations for object recognition and in producing feature representations that predict neural data.

Original: Learning invertible latents.
Revised: Learning invertible latents.

Original: In deep learning applications, invertibility is mostly discussed in generative neural networks as a constraint to learn a prior to address applications in signals and systems such as image de-noising, signal compression and image reconstruction from few and noisy measurements or to be able to reconstruct or modify real images.
Revised: In deep learning applications, invertibility is mostly discussed in generative neural networks as a constraint to learn a prior to address applications in signals and systems such as image de-noising, signal compression and image reconstruction from few and noisy measurements or to be able to reconstruct or modify real images.

Original: Usually invertibility is implemented by carefully designing dedicated architectures (Jacobsen et al., 2018b; Chen et al., 2019).
Revised: Usually invertibility is implemented by carefully designing dedicated architectures (Jacobsen et al., 2018b; Chen et al., 2019).

Original: However, recently it has been shown it can be implemented in standard feedforward ANNs when they undergo training for adversarial robustness (Engstrom et al., 2019b;c).
Revised: However, recently it has been shown it can be implemented in standard feedforward ANNs when they undergo training for adversarial robustness (Engstrom et al., 2019b;c).

Original: These works showed empirically that adversarially robust training encourages invertibility as linear interpolation between classes (e.g., cat to dog) results in semantically smooth image-to-image translation (Engstrom et al., 2019b) as opposed to blurry image sequences produced by standard ANNs.
Revised: These works showed empirically that adversarially robust training encourages invertibility as linear interpolation between classes (e.g., cat to dog) results in semantically smooth image-to-image translation (Engstrom et al., 2019b) as opposed to blurry image sequences produced by standard ANNs.

Original: We reasoned that robust networks which encourage invertibility may also lead to straightening as this a property that would be related to improved invertibility of a network, so we sought to extend prior work and study the behavior of robustly trained networks specifically in the domain of natural movies.
Revised: We reasoned that robust networks which encourage invertibility may also lead to straightening as this a property that would be related to improved invertibility of a network, so we sought to extend prior work and study the behavior of robustly trained networks specifically in the domain of natural movies.

Original: We report on how these networks straighten natural movies in their features spaces and can invertibly reproduce movie frames in a natural sequence. 
Revised: We report on how these networks straighten natural movies in their features spaces and can invertibly reproduce movie frames in a natural sequence. C u r v a t u r e c h a n g e ( d e g r ee s r e : p i x e l s )

Original:  M ETHODS 3. B ASELINE MODELS
Revised:  M ETHODS 3. B ASELINE MODELS

Original: We consider the class of feedforward convolutional neural networks, typically restricting to the
Revised: We consider the class of feedforward convolutional neural networks, typically restricting to the

Original: ResNet-50 (He et al., 2015) architecture trained on ImageNet for the main analyses.
Revised: ResNet-50 (He et al., 2015) architecture trained on ImageNet for the main analyses.

Original: Baseline networks (not trained for robustness) include a supervised ResNet-50/ResNet-101/ResNet-152, and C u r v a t u r e c h a n g e ( d e g r ee s r e : p i x
Revised: Baseline networks (not trained for robustness) include a supervised ResNet-50/ResNet-101/ResNet-152, and self-supervised (Barlowtwins (Zbontar et al., 2021)).

Original: e l s )
Revised: We trained ResNet-50 for imagenet classification without augmentations and with extensive augmentations (Chen et al., 2020), labeled as SupNoAugm and SupMocoAugm , respectively. We also consider Voneresnet (biological V1 front-end (Dapello et al., 2020)) and ResNet-50 trained as a base network for action recognition (Chen et al., 2021) but include these as separate examples in the Appendix since they use a modified architecture.

Original: Non-robust Neural Networks Robust Neural Networks  - - -
Revised: repository in random smoothing without probabilistic inference [Table]

Original: 3. M ODELS TRAINED FOR ROBUSTNESS
Revised: 3. M ODELS TRAINED FOR ROBUSTNESS

Original: We consider two forms of models trained for minimizing a classification loss L ce in the face of input perturbations δ ∈ R h × w × c subject to constraints on the overall magnitude of perturbations in the input space, where x , y , θ are the network input, output and classifier parameters, respectively:
Revised: We consider two forms of models trained for minimizing a classification loss L ce in the face of input perturbations δ ∈ R h × w × c subject to constraints on the overall magnitude of perturbations in the input space, where x , y , θ are the network input, output, and classifier parameters, respectively:

Original: [Equation]
Revised: [Equation]

Original: In adversarially trained networks, projected gradient descent from the output space finds maximal directions of perturbation in the input space limited to length ϵ  and training entails minimizing the effect of these perturbation directions on the network’s output (Madry et al., 2018).In random [Table] N a t u r a l
Revised: In adversarially trained networks, projected gradient descent from the output space finds maximal directions of perturbation in the input space limited to length ϵ , and training entails minimizing the effect of these perturbation directions on the network’s output (Madry et al., 2018).

Original: M o v i e

Original: [Figure]

Original: t t-2 t-4 tR S

Original: A T

Original: S t a n d a r d

Original: P i x e l s ....

Original: Invertibility of Representations

Original: SS I

Original: M t o

Original: R e a l M o v i e F r a m e s 0. smoothing (Lecuyer et al., 2018; Cohen et al., 2019), a supervised network is trained but in the face of Gaussian noise added to the input space as the base classifier before performing a probabilistic inference.
Revised: In random smoothing (Lecuyer et al., 2018; Cohen et al., 2019), a supervised network is trained but in the face of Gaussian noise added to the input space as the base classifier before performing a probabilistic inference.

Original: In this work, we only use the representations as learned in base classifiers without the probabilistic inference.
Revised: In this work, we only use the representations as learned in base classifiers without the probabilistic inference.

Original: The perturbations in the base classifiers δ thus can follow:
Revised: The perturbations in the base classifiers δ thus can follow:

Original: [Equation]
Revised: [Equation]

Original: These defenses to input noise have different motivations.
Revised: These defenses to input noise have different motivations.

Original: Adversarial robustness provides defense against white box attacks whereas random smoothing is protecting against general image corruptions.
Revised: Adversarial robustness provides defense against white box attacks whereas random smoothing is protecting against general image corruptions.

Original: However, prior work has suggested a connection between corruption robustness and adversarial robustness (Ford et al., 2019).
Revised: However, prior work has suggested a connection between corruption robustness and adversarial robustness (Ford et al., 2019).

Original: Theoretically, random smoothing leads to certified robustness (Cohen et al., 2019) and trains a condition of invertible networks (Jacobsen et al., 2018a), while adversarial robustness has been shown empirically to lead to invertible latent representations in networks (Engstrom et al., 2019b).
Revised: Theoretically, random smoothing leads to certified robustness (Cohen et al., 2019) and trains a condition of invertible networks (Jacobsen et al., 2018a), while adversarial robustness has been shown empirically to lead to invertible latent representations in networks (Engstrom et al., 2019b).

Original: 3. R EPRESENTATIONAL M ETRICS
Revised: 3. R EPRESENTATIONAL M ETRICS

Original: Representational straightening estimates the local curvature c in a given representation r of a sequence of images (natural or artificial) of length N , C seq : { x t 1 , x t 2 , ..., x t N } as the angle between vectors connecting nearby frames, and these local estimates are averaged over the entire movie sequence for the overall straightening in that representational trajectory (same as (H´enaff et al., 2019)):c t = arccos (cid:18) r t − r t −∥ r t − r t − 1 ∥ · r t +1 − r t ∥ r t +1 − r t ∥ (cid:19) , C seq = 1 N N − 1 (cid:88) t =
Revised: Representational straightening estimates the local curvature c in a given representation r of a sequence of images (natural or artificial) of length N , C seq : { x t 1 , x t 2 , ..., x t N } as the angle between vectors connecting nearby frames, and these local estimates are averaged over the entire movie sequence for the overall straightening in that representational trajectory (same as (H´enaff et al., 2019)):

Original: c t (3)
Revised: [Equation]

Original: Lower curvature (angle between neighboring vectors) indicates a straighter trajectory, and in the results we generally reference curvature values to the curvature in the input space (i.e., straightening relative to pixel space).
Revised: Lower curvature (angle between neighboring vectors) indicates a straighter trajectory, and in the results we generally reference curvature values to the curvature in the input space (i.e., straightening relative to pixel space).

Original: This metric has been utilized in neuroscience showing that humans tend to represent nearby movie frames in a straightened manner relative to pixels (H´enaff et al., 2019).
Revised: This metric has been utilized in neuroscience showing that humans tend to represent nearby movie frames in a straightened manner relative to pixels (H ´ enaff et al., 2019).

Original: This curvature metric is also closely related to objectives used in efforts to train models with equivariance by linearizing natural transformations in the world as an alternative to standard networks trained for invariant object classification (Goroshin et al., 2015; Sabour et al., 2017).
Revised: This curvature metric is also closely related to objectives used in efforts to train models with equivariance by linearizing natural transformations in the world as an alternative to standard networks trained for invariant object classification (Goroshin et al., 2015; Sabour et al., 2017).

Original: Expansion.
Revised: Expansion.

Original: We define the radius of a sequence of images from a movie clip as the radial size of the minimum covering hyper-sphere circumscribing all points representing the frames in r (G¨artner, 1999).
Revised: We define the radius of a sequence of images from a movie clip as the radial size of the minimum covering hyper-sphere circumscribing all points representing the frames in r (G¨artner, 1999).

Original: We use this measure to supplement geometrical characterization of a movie sequence in pixel space and in a model’s representational spaces.
Revised: We use this measure to supplement geometrical characterization of a movie sequence in pixel space and in a model’s representational spaces.

Original: Like representational straightening values, expansion values for models in the main text are referenced to the radius measured in pixel space or to the radius measure for the same layer in a baseline network by simply dividing by those references.
Revised: Like representational straightening values, expansion values for models in the main text are referenced to the radius measured in pixel space or to the radius measure for the same layer in a baseline network by simply dividing by those references.

Original: We used mini-ball, a publicly available python package based on (G¨artner, 1999) to measure radius of the covering hyper-sphere.
Revised: We used mini-ball, a publicly available python package based on (G¨artner, 1999) to measure radius of the covering hyper-sphere.

Revised:  R ESULTS 4. R OBUST ANN S EXHIBIT REPRESENTATIONAL STRAIGHTENING

Revised: With insights from connections to invertibility (see Figure 1), we hypothesized representational straightening of movie trajectories could be present in robustly trained neural networks.

Revised: We took

Original: N e u r a lp r
Revised: N a t u r a l

Original: e d i c i t i v it y ( Ve x p l a i n ed v a r i a [Figure]
Revised: M o v i e

Original: [Figure]
Revised: [Figure]

Original: Primary visual cortex

Original: r=-0.

Original: represents a layer in ResNet50 trained under different loss function (color codes same as left).

Original: Higher representational straightening (negative curvature change) associates with higher V1 predictivity.

Original: Intriguingly, the highest V1 predictivity corresponds to layers that exhibit comparable straightening to that measured from V1 neurons ( − 10 ◦ on average) (H´enaff et al., 2021). Explained variance is noise-corrected and computed as in (Schrimpf et al., 2018) .  R ESULTS 4. R OBUST ANN SEXHIBIT REPRESENTATIONAL STRAIGHTENING With insights from connections to invertibility (see Figure 1), we hypothesized representational straightening of movie trajectories could be present in robustly trained neural networks.
Revised: t t-2 t-4 tR S

Original: We took the same movie stimuli publicly available (H´enaff et al., 2019)(A.4.1, Figure 12) and the same metrics, and we tested the same architecture, ResNet50 (He et al., 2015)) trained under different loss functions Table 1 to perform controlled head-to-head comparisons.
Revised: A T S t a n d a r d P i x e l s .... SS I M t o

Revised: R e a l

Revised: M o v i e F r a m e s 0.

Revised: sequence in studies of straightening (H ´ enaff et al.,

Revised: , as opposed sequence which the same movie stimuli publicly available (H ´ enaff et al., 2019)(A.4.1, Figure 12) and the same metrics, and we tested the same architecture, ResNet50 (He et al., 2015)) trained under different loss functions Table 1 to perform controlled head-to-head comparisons.

Original: Figure 2 shows representational straightening of natural movies measured in layers of ResNet50 trained under AT (Engstrom et al., 2019a) and RS (Cohen et al., 2019) at different adversarial attack or noise levels, respectively.
Revised: Figure 2 shows representational straightening of natural movies measured in layers of ResNet50 trained under AT (Engstrom et al., 2019a) and RS (Cohen et al., 2019) at different adversarial attack or noise levels, respectively.

Original: Robust neural networks in contrast to other ANNs decreased the curvature of natural movies.
Revised: Robust neural networks in contrast to other ANNs decreased the curvature of natural movies.

Original: Straightening for artificial sequences as measured in (H´enaff et al., 2019)
Revised: Straightening for artificial sequences as measured in (H´enaff et al., 2019)

Original: (A.1, Figure 7) and other models (A.2, Figures 9 and 8) are provided in Appendix.
Revised: (A.1, Figure 7) and other models (A.2, Figures 9 and 8) are provided in Appendix.

Original: Importantly, although most models, whether a standard ResNet-50 or one with a V1-like front-end, may display an initial dip in curvature for natural movies in the very earliest layers, this is not sustained in feature representations of later layers except for robustly trained networks (A.2, Figure 9 vs. A.1, Figure 7) and those trained on action recognition from temporally instructed training, which we include here as a proxy for a movielike training though its feedforward architecture deviates from a ResNet50 by additional temporal processing components (A.2, Figure 8).
Revised: Importantly, although most models, whether a standard ResNet-50 or one with a V1-like front-end, may display an initial dip in curvature for natural movies in the very earliest layers, this is not sustained in feature representations of later layers except for robustly trained networks (A.2, Figure 9 vs. A.1, Figure 7) and those trained on action recognition from temporally instructed training, which we include here as a proxy for a movielike training though its feedforward architecture deviates from a ResNet50 by additional temporal processing components (A.2, Figure 8).

Original: Perceptual Straightening measured as invertibility of latent representations.
Revised: Perceptual Straightening measured as invertibility of latent representations.

Original: Next, we sought to empirically test how well robust networks can invert natural movies given that they contain linearized feature representation of movie frames in their high level feature spaces and given the general conceptual benefit of linearity for invertibility Figure 1.
Revised: Next, we sought to empirically test how well robust networks can invert natural movies given that they contain linearized feature representation of movie frames in their high level feature spaces and given the general conceptual benefit of linearity for invertibility Figure 1.

Original: We measured invertibility of each model on the same movie sequences used for measuring straightening as follows.
Revised: We measured invertibility of each model on the same movie sequences used for measuring straightening as follows.

Original: We linearly interpolated between latent representation of the first and last frame of each movie and used the same procedure as that used previously in (Engstrom et al., 2019b;a) to obtain the pixel-space correspondence of those interpolated representations. 
Revised: We linearly interpolated between latent representation of the first and last frame of each movie and used the same procedure as that used previously in (Engstrom et al., 2019b;a) to obtain the pixel-space correspondence of those interpolated representations. Whereas those generated pseudo-frames can be assessed by for their pixel-by-pixel distance to the actual movie frame, we chose a metric, Structural Similarity In- N e u r a l p r e

Revised: d i c i t i v i

Original: Whereas those generated pseudo-frames can be assessed by for their pixel-by-pixel distance to the actual movie frame, we chose a metric, Structural Similarity Index Measure (SSIM (Wang et al., 2004)), that utilizes intermediate-level statistics motivated from biological vision and putatively more related to some aspects of human perception than simple pixel space correspondence.
Revised: t y ( Ve x p l a i n e d v a r i a n c e ) [Figure] r=-0. Measure (SSIM (Wang et al., 2004)), that utilizes intermediate-level statistics motivated from biological vision and putatively more related to some aspects of human perception than simple pixel space correspondence.

Original: Figure 3 shows an example of such inverted frames for standard ResNet50,
Revised: Figure 3 shows an example of such inverted frames for standard ResNet50,

Original: RS ( L 2 : σ 2 = 0 . 5 ) and AT ( L 2 :
Revised: RS ( L 2 : σ 2 = 0 . 5 ) and AT ( L 2 :

Original: σ 2 = 3 ), and a summary of average measured invertibility using the SSIM metric on pseudo-frames from each model.
Revised: σ 2 = 3 ), and a summary of average measured invertibility using the SSIM metric on pseudo-frames from each model.

Original: As expected, inline with the findings of previous work (Engstrom et al., 2019b), AT models scored relatively higher on invertibility of frames than a baseline discriminative model.
Revised: As expected, inline with the findings of previous work (Engstrom et al., 2019b), AT models scored relatively higher on invertibility of frames than a baseline discriminative model.

Original: However, what had not been previously shown is that RS models, using merely the benefits of their robustness to noisy augmentation (base classifier on top of learned representation; no probabilistic inference), also exhibit higher invertibility scores compared to standard trained models.
Revised: However, what had not been previously shown is that RS models, using merely the benefits of their robustness to noisy augmentation (base classifier on top of learned representation; no probabilistic inference), also exhibit higher invertibility scores compared to standard trained models.

Original: Invertibility scores were consistently improved in RS and AT models across a variety of movies tested including those with relatively stationary textures and not just dynamic objects (see A.4.4, Figure 13 for further examples and A.4.3, Table 3 for scores across all 11 movies).
Revised: Invertibility scores were consistently improved in RS and AT models across a variety of movies tested including those with relatively stationary textures and not just dynamic objects (see A.4.4, Figure 13 for further examples and A.4.3, Table 3 for scores across all 11 movies).

Original: Thus, RS models along with AT models exhibit invertibility of representations for movie frames which further demonstrates their ability to support perceptual straightening of natural movies in their highest layers that may be functionally similar to perceptual straightening previously measured from human subjects (H ´ enaff et al., 2019).
Revised: Thus, RS models along with AT models exhibit invertibility of representations for movie frames which further demonstrates their ability to support perceptual straightening of natural movies in their highest layers that may be functionally similar to perceptual straightening previously measured from human subjects (H´enaff et al., 2019).

Original: 4. R ANDOM SMOOTHING AND ADVERSARIAL TRAINING IN EXPLAINING NEURAL REPRESENTATIONS IN THE PRIMATE VISUAL SYSTEM
Revised: 4. R ANDOM SMOOTHING AND ADVERSARIAL TRAINING IN EXPLAINING NEURAL REPRESENTATIONS IN THE PRIMATE VISUAL SYSTEM

Original: Robustness to noise as a bio-plausible mechanism underlying straightening in primary visual cortex.
Revised: Robustness to noise as a bio-plausible mechanism underlying straightening in primary visual cortex.

Original: As shown above, straightening which is a constraint for brain-like representations in visual cortex manifests in robust neural networks.
Revised: As shown above, straightening which is a constraint for brain-like representations in visual cortex manifests in robust neural networks.

Original: Both classes of RS and AT training for robustness to
Revised: Both classes of RS and AT training for robustness to

Original: L 2 norm generate straightened representations of movie sequences.
Revised: L 2 norm generate straightened representations of movie sequences.

Original: However, to distinguish among models of object recognition, we can measure how well they explain variance in patterns of neural activity elicited in different visual cortical areas.
Revised: However, to distinguish among models of object recognition, we can measure how well they explain variance in patterns of neural activity elicited in different visual cortical areas.

Original: Here, for all neural comparisons in our analyses, we measured the Brain-Score (Schrimpf et al., 2018) using the publicly available online resource to assess the similarity to biological vision of each model, which is a battery of tests comparing models against previously collected data from the primate visual system (see Brain-Score.org).
Revised: Here, for all neural comparisons in our analyses, we measured the Brain-Score (Schrimpf et al., 2018) using the publicly available online resource to assess the similarity to biological vision of each model, which is a battery of tests comparing models against previously collected data from the primate visual system (see Brain-Score.org).

Original: We found that RS and AT models provided a better model of V1 (in terms of explained variance) compared to non-robust models Figure 4.
Revised: We found that RS and AT models provided a better model of V1 (in terms of explained variance) compared to non-robust models Figure 4.

Original: On other benchmarks, as we go up the ventral stream hierarchy from V  to IT again, keeping the layer assignment fixed across models for proper comparison, we observed a decrease in explainability of robust models (A.3, Figure 11), in part presumably because robust models have lower object classification performance which is known to drive fits in higher brain areas like V4 and IT supporting object recognition (Yamins et al., 2014).
Revised: On other benchmarks, as we go up the ventral stream hierarchy from V  to IT again, keeping the layer assignment fixed across models for proper comparison, we observed a decrease in explainability of robust models (A.3, Figure 11), in part presumably because robust models have lower object classification performance which is known to drive fits in higher brain areas like V4 and IT supporting object recognition (Yamins et al., 2014).

Original: Previous work (Dapello et al., 2020; Kong et al., 2022) linked adversarial robustness in models to their higher Brain-Score for
Revised: Previous work (Dapello et al., 2020; Kong et al., 2022) linked adversarial robustness in models to their higher Brain-Score for

Original: V1, but we found that it may not be specifically driven by adversarial robustness per se, rather ( L 2 ) noise robustness is also sufficient (as in base classifiers of RS tested here).
Revised: V1, but we found that it may not be specifically driven by adversarial robustness per se, rather ( L 2 ) noise robustness is also sufficient (as in base classifiers of RS tested here).

Original: More broadly, looking at neural fits across all models and their layers, we find that straightening in a particular model-layer correlates with improved explanatory power of variance in cortical area V1 (Figure 4, middle panel, each dot is a layer from a model), being even more strongly predictive than robustness of the overall model (A3, Figure 10).
Revised: More broadly, looking at neural fits across all models and their layers, we find that straightening in a particular model-layer correlates with improved explanatory power of variance in cortical area V1 (Figure 4, middle panel, each dot is a layer from a model), being even more strongly predictive than robustness of the overall model (A3, Figure 10).

Original: The level of straightening reached by best fitting layers of RS and AT models was comparable to the 10 degree straightening estimated in macaque V1 neural populations (black dashed reference line in Figure 4).
Revised: The level of straightening reached by best fitting layers of RS and AT models was comparable to the 10 degree straightening estimated in macaque V1 neural populations (black dashed reference line in Figure 4).

Original: This complements the fact that robust models peak near thedegree straightening measured in perception (Figure 2), suggesting that robust models can achieve a brain-like level of straightening to V1 and perception.
Revised: This complements the fact that robust models peak near thedegree straightening measured in perception (Figure 2), suggesting that robust models can achieve a brain-like level of straightening to V1 and perception.

Original: Does the geometry of movie frame representations in pixel space dictate straightening in downstream representations?
Revised: Does the geometry of movie frame representations in pixel space dictate straightening in downstream representations?

Original: The connection between two properties of the same representation manifold, robustness to independently sampled noise and straightened trajectories of smooth input temporal sequences, is not immediately clear.
Revised: The connection between two properties of the same representation manifold, robustness to independently sampled noise and straightened trajectories of smooth input temporal sequences, is not immediately clear.

Original: Because robustness is achieved by adding noise bounded by a norm ( L 2 , L 2 , or L ∞ ) in pixel space, a natural question is whether the radius of the bounding hyper-sphere of the frames of the tested movies in pixel space (see Expansion in Methods) was correlated with the measured straightening in feature space in each layer of the robustly trained models (Figure 5; also see A.5, Figure 14).
Revised: Because robustness is achieved by adding noise bounded by a norm ( L 2 , L 2 , or L ∞ ) in pixel space, a natural question is whether the radius of the bounding hyper-sphere of the frames of the tested movies in pixel space (see Expansion in Methods) was correlated with the measured straightening in feature space in each layer of the robustly trained models (Figure 5; also see A.5, Figure 14).

Original: We found, however, that there seemed to be different mechanisms at play for RS versus AT in terms of achieving straightening.
Revised: We found, however, that there seemed to be different mechanisms at play for RS versus AT in terms of achieving straightening.

Original: RS models showed (small but) positive correlations, which means the smaller the ball containing all the frames of the movie in input space, the larger the straightening effect for the representations of frames of that movie in the model.
Revised: RS models showed (small but) positive correlations, which means the smaller the ball containing all the frames of the movie in input space, the larger the straightening effect for the representations of frames of that movie in the model.

Original: While in AT models we see the opposite (negative) or no correlation.
Revised: While in AT models we see the opposite (negative) or no correlation.

Original: These divergent patterns underscore differences between these models and suggest that geometric size in pixel space is not strongly constraining the degree to which a movie can be straightened.
Revised: These divergent patterns underscore differences between these models and suggest that geometric size in pixel space is not strongly constraining the degree to which a movie can be straightened.

Original: Geometry of movie frame representations in feature space is relevant for capturing neural representations in V1 Between different RS models tested on different input noise levels, RS
Revised: Geometry of movie frame representations in feature space is relevant for capturing neural representations in V1 Between different RS models tested on different input noise levels, RS

Original: L 2 : σ 2 = 0 .
Revised: L 2 : σ 2 = 0 .

Original: 5 stands out as it gives a better model of V1 than those using smaller or larger magnitude input noise (Figure 4).
Revised: 5 stands out as it gives a better model of V1 than those using smaller or larger magnitude input noise (Figure 4).

Original: For this model, we found that in addition to its intermediate level of straightening, the expansion score of movie frames, which is the radial size in its representation normalized to size in the same layer of a baseline ResNet50, was highest compared to theother RS models (Figure 6, middle panel; measures are referenced to layers in a standard ResNet to highlight relative effect of robustness training rather than effects driven by hierarchical layer). This demonstrates a potential trade-off between improving straightening in a representation while avoiding too much added contraction of movies by robust training relative to standard training. This balance seems to be best achieved for σ 2 = 0 . 5 , where we also see the significantly higher predic-
Revised: For this model, we found that in addition to its intermediate level of straightening, the expansion score of movie frames, which is the radial size in its representation normalized to size in the same layer of a baseline ResNet50, was highest compared to the

Original: ∆ S t r a
Revised: ∆ S t r a

Original: i g h t e n i n g ( r e : S t a n d a r d R)
Revised: i g h t e n i n g ( r e : S t a n d a r d R)

Original: ∆
Revised: ∆

Original: E x p a n s i o n ( r e : S t a n d a r d
Revised: E x p a n s i o n ( r e : S t a n d a r d

Original: R 50 )
Revised: R 50 )

Original: ∆ B r a i nS c
Revised: ∆ B r a i nS c

Original: o r e
Revised: o r e

Original: V 1 ( r e :
Revised: V 1 ( r e :

Original: S t a n d a r d R 50 ) tivity of V1 cortical data (Figure 6, 
Revised: S t a n d a r d R 50 ) other This demonstrates a potential trade-off between improving straightening in a representation while avoiding too much added contraction of movies by robust training relative to standard training. This balance seems to be best achieved for σ 2 = 0 . 5 , where we also see the significantly higher predictivity of V1 cortical data (Figure 6, right panel).

Original: The best  model  little contraction 
Revised: The best AT model also shows little contraction of movies coupled with high straightening (A.5, 15).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have demonstrated novel properties of robust neural networks in how they represent natural movies.
Revised: We have demonstrated novel properties of robust neural networks in how they represent natural movies.

Original: Conceptually, this work establishes a seemingly surprising connection between disparate ideas, robust discriminative networks trained on static images on one hand, to work learning to linearize by training on natural movies, on the other.
Revised: Conceptually, this work establishes a seemingly surprising connection between disparate ideas, robust discriminative networks trained on static images on one hand, to work learning to linearize by training on natural movies, on the other.

Original: These modeling paths could both result in linearized, or straightened, natural movie representations (Figure 1).
Revised: These modeling paths could both result in linearized, or straightened, natural movie representations (Figure 1).

Original: From a machine learning perspective, the invertibility and concomitant representational straightening of robust networks suggests that they learn explainable representations of natural movie statistics.
Revised: From a machine learning perspective, the invertibility and concomitant representational straightening of robust networks suggests that they learn explainable representations of natural movie statistics.

Original: Biologically, the emergence of straightening in these networks as well as their ability to better explain V1 data than baselines relatively lacking in straightening Figure 4 provides new insights into potential neural mechanisms for previously difficult to explain brain phenomena.
Revised: Biologically, the emergence of straightening in these networks as well as their ability to better explain V1 data than baselines relatively lacking in straightening Figure 4 provides new insights into potential neural mechanisms for previously difficult to explain brain phenomena.

Original: Biological constraints could lend parsimony to selecting among models, each with a different engineering goal.
Revised: Biological constraints could lend parsimony to selecting among models, each with a different engineering goal.

Original: On face, RS by virtue of utilizing Gaussian noise instead of engineered noise gains traction over adversarial training as a more simple, and powerful way of achieving robustness in
Revised: On face, RS by virtue of utilizing Gaussian noise instead of engineered noise gains traction over adversarial training as a more simple, and powerful way of achieving robustness in

Original: ANNs, which is inline with a long history of probabilistic inference in visual cortex of humans (Pouget et al., 2013).
Revised: ANNs, which is inline with a long history of probabilistic inference in visual cortex of humans (Pouget et al., 2013).

Original: Indeed, looking across the range of robust models tested, the best fitting model of V1 was not necessarily the most robust but tended toward more straightened representations that also showed the least contracted representations – consistent with a known dimensionality expansion from the sensory periphery to V1 in the brain (Field, 1994).
Revised: Indeed, looking across the range of robust models tested, the best fitting model of V1 was not necessarily the most robust but tended toward more straightened representations that also showed the least contracted representations – consistent with a known dimensionality expansion from the sensory periphery to V1 in the brain (Field, 1994).

Original: Future work exploring a wider variety of robustness training in conjunction with more bioplausible architectures, objectives, and training diets may yet elucidate the balance of factors contributing to biological vision.
Revised: Future work exploring a wider variety of robustness training in conjunction with more bioplausible architectures, objectives, and training diets may yet elucidate the balance of factors contributing to biological vision.

Original: At the same time, our work does not directly address how straightened representations in the visual system may or may not be utilized to influence downstream visual perception and behavior, and this connection is an important topic for future work.
Revised: At the same time, our work does not directly address how straightened representations in the visual system may or may not be utilized to influence downstream visual perception and behavior, and this connection is an important topic for future work.

Original: On the one hand, for supporting dynamical scene perception, behaviors that predict (extrapolate) or postdict (interpolate) scene properties over time (e.g., object position) may be supported by straightened natural movie representations.
Revised: On the one hand, for supporting dynamical scene perception, behaviors that predict (extrapolate) or postdict (interpolate) scene properties over time (e.g., object position) may be supported by straightened natural movie representations.

Original: Indeed, both explanations, prediction and postdiction, have been invoked to account for psychophysical phenomena like the flash-lag illusion which present an interesting test case of how the brain processes complex stimuli over time (Eagleman & Sejnowski, 2000).
Revised: Indeed, both explanations, prediction and postdiction, have been invoked to account for psychophysical phenomena like the flash-lag illusion which present an interesting test case of how the brain processes complex stimuli over time (Eagleman & Sejnowski, 2000).

Original: However, even for relatively stationary scenes such as those containing textures, we observed benefits for straightening and invertibility in robustly trained networks (see A.4, Tables 2 and 3).
Revised: However, even for relatively stationary scenes such as those containing textures, we observed benefits for straightening and invertibility in robustly trained networks (see A.4, Tables 2 and 3).

Original: Further work is needed to explore how spatially local versus global features in the presence of simple versus complex motion are affected in their relative straightening by model training.
Revised: Further work is needed to explore how spatially local versus global features in the presence of simple versus complex motion are affected in their relative straightening by model training.

Revised: A CKNOWLEDGMENTS

Revised: This work was supported by a Klingenstein-Simons fellowship, Sloan Foundation fellowship, and

Revised: Grossman-Kavli Scholar Award as well as a NVIDIA GPU grant and was performed using the

Revised: Columbia Zuckerman Axon GPU cluster.

Revised: We thank all three reviewers for their constructive feedback that led to an improved final version of the paper.

Original: A.STRAIGHTENING IN OTHER ARCHITECTURES
Revised: A.

Revised: STRAIGHTENING IN OTHER ARCHITECTURES

Original: [Figure]
Revised: [Figure]

Original: N a t [Figure]
Revised: A r t i f i c i a l et al., 2021)

Original: A r t i f i c i a l were trained 
Revised: They were trained on video for

Original: Although these  were not trained for straightening or predicting the next
Revised: Although these models were not trained for straightening or predicting the next

Original: , they exhibit small but measurable straightening for 
Revised: , they exhibit small but measurable straightening for natural movies.

Original: However, the curvatures for artificial sequences were not increased as much as curvature increase for artificial sequences in robust neural networks Figure 7)[Figure]
Revised: However, the curvatures for artificial sequences were not increased as much as curvature increase for artificial sequences in robust neural networks (Figure 7)

Original: [Figure]
Revised: [Figure]

Original: PREDICITIVITY

Original: r=0.28 (p=2.9e-12) r=0.06 (p=0.2)  0.

Original: [Figure]
Revised: [Figure]

Original:  e u r a l p r ed i c
Revised: N e u r a l p r e

Original:  i t i v it y
Revised: d i c i t i v i

Original:  ( Ve x p l a i n e d v a r i a n c 
Revised: t y ( Ve x p l a i n e d v a r i a n c e )

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: M OVIE CHARACTERISTICS AND ADDITIONAL INTERPOLATION EXAMPLES A.4. M OVIES w a t e r c a r n e g i e - d a m w a l k i n g d o g v ill e e g o m o ti o n c
Revised: w a t e r c a r n e g i e - d a m w a l k i n g d o g v ill e e g o m o t

Original:  h
Revised: i o n c h

Original: i r o n o m o
Revised: i r o n o m o

Original: u s b ee s l e a v e s - w i n d s m il e c h
Revised: u s b ee s l e a v e s - w i n d s m il e c h

Original: i r o n o m u s p r a
Revised: i r o n o m u s p r a

Original: i r i e T e r
Revised: i r i e T e r

Original: A.4. T ABLE FOR AVERAGE STRAIGHTENING 
Revised: [Table] A.

Original: [Table]
Revised: [Table]

Original: I NVERTIBILITY MEASURE FOR EACH MOVIE
Revised: I NVERTIBILITY MEASURE FOR EACH MOVIE

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: 0.34 RN
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 0.31 RN50 NoAugm 0.

Original: 0.32 RN 0.
Revised: 32 RN 0.

Revised: [Table]

Original: 0.35 RN 0.
Revised: [Table] 35 RN [Table]

Original:  0.36 AT L 2 : ϵ =0.
Revised: [Table] [Table] [Table] 0.36 AT L 2 : ϵ =0.

Original: 0.59 AT L ∞ : ϵ =0.
Revised: 0.59 AT L ∞ : ϵ =0.

Original: 0.56 AT L ∞ : ϵ =0.
Revised: 0.56 AT L ∞ : ϵ =0.

Original: 0.62 RS L 2 : ϵ = 0 .
Revised: 0.62 RS L 2 : ϵ = 0 .

Original: 0.42 RS L 2 : ϵ = 0 .0.
Revised: 0.42 RS L 2 : ϵ = 0 .0.

Original: 0.46 RS L 2 : ϵ =0.
Revised: 0.46 RS L 2 : ϵ =0.

Original: [Figure]
Revised: [Figure]

Revised: v a

Original: [Figure]
Revised: [Figure]

Revised: u r e ( r e : p i x

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. R EPRODUCIBILITY INFORMATION
Revised: A. R EPRODUCIBILITY INFORMATION

Original: Almost all data (models, movies, and metrics) used in this work are publicly available and we provided references to them in the text (for instance see Table 1).
Revised: Almost all data (models, movies, and metrics) used in this work are publicly available and we provided references to them in the text (for instance see Table 1).

Original: We will release the code to reproduce the main results in this work and we provide pointers to the publicly available resources used in this work as listed below.
Revised: We will release the code to reproduce the main results in this work and we provide pointers to the publicly available resources used in this work as listed below.

Original: Movies and images.
Revised: Movies and images.

Original: We used the same movies used in the original studies on human perception and monkey primary visual cortex (H´enaff et al., 2019; 2021) which are available from first author
Revised: We used the same movies used in the original studies on human perception and monkey primary visual cortex (H´enaff et al., 2019; 2021) which are available from first author

Original: Github as referenced in their papers.
Revised: Github as referenced in their papers.

Original: Images used to measure the clean accuracy and robust accuracy were taken from ImageNet validation set.
Revised: Images used to measure the clean accuracy and robust accuracy were taken from ImageNet validation set.

Original: Models.
Revised: Models.

Original: All the models used in this study were from ResNet family and checkpoints for the main robust models are publicly available as references in the main text (Table 1).
Revised: All the models used in this study were from ResNet family and checkpoints for the main robust models are publicly available as references in the main text (Table 1).

Original: The checkpoints for the only two custom trained models (supervised with no augmentions and supervised with Moco augmentation) will be made publicly available along with the code.
Revised: The checkpoints for the only two custom trained models (supervised with no augmentions and supervised with Moco augmentation) will be made publicly available along with the code.

Original: Neural predictivity metric.
Revised: Neural predictivity metric.

Original: We used brain-score, which is a publicly available benchmark to evaluate how well a model predicts variance in neural data ( [Table] [Table]
Revised: We used brain-score, which is a publicly available benchmark to evaluate how well a model predicts variance in neural data (Schrimpf et al., 2018).