Original: Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V 
Revised: Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V

Original: Abstract
Revised: Abstract

Original: Modern high-scoring models of vision in the brain score competition do not stemfrom Vision Transformers.
Revised: Modern high-scoring models of vision in the brain score competition do not stem from Vision Transformers.

Original: However, in this short paper, we provide evidenceagainst the unexpected trend of Vision Transformers (ViT) being not perceptuallyaligned with human visual representations by showing how a dual-stream Transformer, a CrossViT a la Chen et al.
Revised: However, in this short paper, we provide evidence against the unexpected trend of Vision Transformers (ViT) being not perceptually aligned with human visual representations by showing how a dual-stream Transformer, a CrossViT a la Chen et al.

Original: (2021), under a joint rotationally-invariantand adversarial optimization procedure yields 2nd place in the aggregate BrainScore 2022 competition  averaged across all visual categories, and currently (March1st, 2022) holds the 1st place for the highest explainable variance of area V4.
Revised: (2021), under a joint rotationallyinvariant and adversarial optimization procedure yields 2nd place in the aggregate Brain-Score 2022 competition (Schrimpf et al., 2020b) averaged across all visual categories, and currently (March 1st, 2022) holds the 1st place for the highest explainable variance of area V4.

Original: In addition, our current Transformer-based model also achieves greater explainable variance for areas V4, IT and Behaviour than a biologically-inspired CNN(ResNet50) that integrates a frontal V1-like computation module (Dapello et al.,2020).
Revised: In addition, our current Transformerbased model also achieves greater explainable variance for areas V4, IT and Behaviour than a biologically-inspired CNN (ResNet50) that integrates a frontal V1-like computation module (Dapello et al., 2020).

Original: Our team was also the only entry in the top-5 that shows a positive rankcorrelation between explained variance per area and depth in the visual hierarchy.
Revised: Our team was also the only entry in the top-5 that shows a positive rank correlation between explained variance per area and depth in the visual hierarchy.

Original: Against our initial expectations, these results provide tentative support for an “Allroads lead to Rome” argument enforced via a joint optimization rule even for nonbiologically-motivated models of vision such as Vision Transformers.
Revised: Against our initial expectations, these results provide tentative support for an “All roads lead to Rome” argument enforced via a joint optimization rule even for non biologicallymotivated models of vision such as Vision Transformers.

Revised: Code is available at https://github.com/williamberrios/BrainScore-Transformers

Original:  Optimizing a CrossViT for Brain-Score
Revised:  Optimizing a CrossViT for Brain-Score

Original: In this short paper we discuss an interesting finding, where amidst the constant debate of thebiological plausibility of Vision Transformers – that have been deemed less biologically plausiblethan convolutional neural networks (as discussed in: URL_1 URL_2, though also see Conwell et al.(2021))
Revised: In this short paper we discuss an interesting finding, where amidst the constant debate of the biological plausibility of Vision Transformers – that have been deemed less biologically plausible than convolutional neural networks (as discussed in: URL_1 URL_2, though also see Conwell et al.

Original: –, we find that when these Transformers are optimized under certain conditions, they mayachieve high explainable variance with regards to many areas in primate vision and surprisinglythe highest score to date for explainable variance in area V4, that still remains a mystery in visualneuroscience (see Pasupathy et al.
Revised: we find that when these Transformers are optimized under certain conditions they may achieve high explainable variance with regards to many areas in primate vision, and surprisingly the highest score to date for explainable variance in area V4, that still remains a mystery in visual neuroscience (see Pasupathy et al.

Original: (2020) for a review).
Revised: (2020) for a review).

Original: Our final model was based on several insights:
Revised: Our final model was based on several insights:

Original: Adversarial-Training : Work by Santurkar et al.
Revised: Adversarial-Training : Work by Santurkar et al.

Original: Engstrom et al.
Revised: Engstrom et al.

Revised: ; Dapello et al.

Original: (2019), has shown thatconvolutional neural networks trained adversarially 1 yield human perceptually-aligned distortionswhen attacked.
Revised: (2020), has shown that convolutional neural networks trained adversarially 1 yield human perceptually-aligned distortions when attacked.

Original: This is an interesting finding, that perhaps extends to vision transformers, but has  though  this one (See Figure 1)– have
Revised: This is an interesting finding, that perhaps extends to vision transformers, but has never been qualitatively tested before though recent works – including this one (See Figure 1)

Original: [Table]
Revised: [Table]

Original:  started to investigate in this direction (Tuli et al., 2021;
Revised: – have started to investigate in this direction (Tuli et al., 2021;

Original: Caro et al., 2020).
Revised: Caro et al., 2020).

Original: Thus we projected thatonce we picked a specific vision transformer architecture, we would train it adversarially.
Revised: Thus we projected that once we picked a specific vision transformer architecture, we would train it adversarially.

Original: Multi-Resolution : Pyramid (Burt & Adelson, 1987; Simoncelli & Freeman, 1995; Heeger & Bergen,1995) approaches have been shown to correlate highly with good models of Brain-Scores (Marqueset al., 2021).
Revised: Multi-Resolution : Pyramid (Burt & Adelson, 1987; Simoncelli & Freeman, 1995; Heeger & Bergen, 1995) approaches have been shown to correlate highly with good models of Brain-Scores (Marques et al., 2021).

Original: We devised that our Transformer had to incorporate this type of processing.
Revised: We devised that our Transformer had to incorporate this type of processing.

Original: Rotation Invariance : Object identification is generally rotationally invariant (depending on thecategory; e.g. not the case for faces (Kanwisher et al., 1998)).
Revised: Rotation Invariance : Object identification is generally rotationally invariant (depending on the category; e.g. not the case for faces (Kanwisher et al., 1998)).

Original: So we implicitly trained our modelto take in different rotated object samples via rotation-based data augmentation.
Revised: So we implicitly trained our model to take in different rotated object samples via rotation-based data augmentation.

Original: This procedure isdifferent from pioneering work of Ecker et al.
Revised: This procedure is different from pioneering work of Ecker et al.

Original: (2019) that explicitly added rotation equivariance to aconvolutional neural network.
Revised: (2019) that explicitly added rotation equivariance to a convolutional neural network.

Original: Localized texture-based computation : Despite the emergence of a global texture-bias in objectrecognition when training Deep Neural Networks (Geirhos et al., 2019) – object recognition is acompositional process (Brendel & Bethge, 2019; Deza et al., 2020).
Revised: Localized texture-based computation : Despite the emergence of a global texture-bias in object recognition when training Deep Neural Networks (Geirhos et al., 2019) – object recognition is a compositional process (Brendel & Bethge, 2019; Deza et al., 2020).

Original: Recently, works in neurosciencehave also suggested that local texture computation is perhaps pivotal for object recognition to eithercreate an ideal basis set from which to represent objects (Long et al., 2018; Jagadeesh & Gardner,2022) and/or encode robust representations (Harrington & Deza, 2022).
Revised: Recently, works in neuroscience have also suggested that local texture computation is perhaps pivotal for object recognition to either create an ideal basis set from which to represent objects (Long et al., 2018; Jagadeesh & Gardner, 2022) and/or encode robust representations (Harrington & Deza, 2022).

Original: After searching for several models in the computer vision literature that resemble a Transformermodel that ticks all the boxes of above, we opted for a CrossViT-18 † (that includes multi-resolution+ local texture-based computation) that was trained with rotation-based augmentations and alsoadversarial training (See Appendix A.3 for exact training details, our best model also used p = 0 . 25grayscale augmentation, though this contribution to model Brain-Score is minimal).
Revised: After searching for several models in the computer vision literature that resemble a Transformer model that ticks all the boxes of above, we opted for a CrossViT-18 † (that includes multi-resolution + local texture-based computation) that was trained with rotation-based augmentations and also adversarial training (See Appendix A.3 for exact training details, our best model also used p = 0 .grayscale augmentation, though this contribution to model Brain-Score is minimal).

Original: Results: Our best performing model (#991) achieved 2nd place in the overall Brain-Scorecompetition as shown in Table 1.
Revised: Results: Our best performing model (#991) achieved 2nd place in the overall Brain-Score (Schrimpf et al., 2020b)) competition as shown in Table 1.

Original: Currently, it holds the first place for the highest explainable variance of area V4 and the second highest score in the IT area.
Revised: Currently, it holds the first place for the highest explainable variance of area V4 and the second highest score in the IT area.

Original: Selected layers usedfrom CrossViT-18 † are shown in Table 2, more information can be seen in Appendix C.
Revised: Selected layers used from CrossViT-18 † are shown in Table 2, more information can be seen in Appendix C.

Original: [Table]
Revised: [Table]

Original: Additionally, in comparison with the biologicallyinspired model (Voneresnet50 + Adv. training), ourmodel achieves greater scores in the IT, V4 and
Revised: Additionally, in comparison with the biologicallyinspired model (Voneresnet50 + Adv. training), our model achieves greater scores in the IT, V4 and

Original: Behavioral benchmarks.
Revised: Behavioral benchmarks.

Original: Critically we notice thatour best performing model (#991) has a positive ρHierarchy coefficient 2 compared to the new state ofthe art model (#1033) and other remaining entries,where this coefficient is negative.
Revised: Critically we notice that our best performing model (#991) has a positive ρHierarchy coefficient 2 compared to the new state of the art model (#1033) and other remaining entries, where this coefficient is negative.

Original: This was an unexpected result that we found as most biologicallydriven models obtain higher Brain-Scores at initialstages of the visual hierarchy (V1) (Dapello et al., 2020), and these scores decrease as a function ofhierarchy with general worse Brain-Scores in the final stages ( e.g. IT).[Table]
Revised: This was an unexpected result that we found as most biologicallydriven models obtain higher Brain-Scores at initial stages of the visual hierarchy (V1) (Dapello et al., 2020), and these scores decrease as a function of hierarchy with general worse Brain-Scores in the final stages ( e.g. IT).

Original: We also investigated the differential effects of rotation invariance and adversarial training used ontop of a pretrained CrossViT-18 † as shown in Table 3.
Revised: We also investigated the differential effects of rotation invariance and adversarial training used on top of a pretrained CrossViT-18 † as shown in Table 3.

Original: We observed that each step independentlyhelps to improve the overall Brain-Score.
Revised: We observed that each step independently [Table] helps to improve the overall Brain-Score.

Original: Interestingly, when both methods are combined, the modeloutperforms the baseline behavioral score by a large margin (+0.062).
Revised: Interestingly, when both methods are combined, the model outperforms the baseline behavioral score by a large margin (+0.062).

Original: Finally, our best model alsoretains a great standard accuracy at ImageNet from its pretrained version.
Revised: Finally, our best model also retains a great standard accuracy at ImageNet from its pretrained version [Table]

Original:  Discussion
Revised:  Discussion

Original: A question from this work that requires further investigation is why a CrossViT-18 † performs so wellat explaining variance in primate area V4 without many iterations of hyper-parameter engineering?
Revised: A question from this work that requires further investigation is why a CrossViT-18 † performs so well at explaining variance in primate area V4 without many iterations of hyper-parameter engineering?

Original: We do not know , and we are currently investigating this.
Revised: We do not know , and we are currently investigating this.

Original: One possibility is that cross-attentionmechanism of the CrossViT-18 † is a proxy for Gramian-like operations that encode local texturecomputation (vs global a la Geirhos et al.
Revised: One possibility is that cross-attention mechanism of the CrossViT-18 † is a proxy for Gramian-like operations that encode local texture computation (vs global a la Geirhos et al.

Original: (2019)) which have been shown to be pivotal for objectrepresentation in humans (Long et al., 2018; Jagadeesh & Gardner, 2022; Harrington & Deza, 2022).
Revised: (2019)) which have been shown to be pivotal for object representation in humans (Long et al., 2018; Jagadeesh & Gardner, 2022; Harrington & Deza, 2022).

Original: However, further experiments are required to verify this hypothesis.
Revised: However, further experiments are required to verify this hypothesis.

Original: Finally, one of our most interesting qualitative results is that the direction of the adversarial attackmade on our highest performing model resembles a distortion class that seems to fool a humanobserver too (Figure 1).
Revised: Finally, one of our most interesting qualitative results is that the direction of the adversarial attack made on our highest performing model resembles a distortion class that seems to fool a human observer too (Figure 1).

Original: In the future we are planning on psychophysically testing this phenomenon.
Revised: In the future we are planning on psychophysically testing this phenomenon.

Original: [Figure]
Revised: [Figure]

Original: A Experimental Setup 
Revised: A Experimental Setup A.1 Dataset

Original: [Figure] A.1 Dataset

Original: [Figure] We used the ImageNet 1k (Deng et al., 2009) dataset for training.
Revised: We used the ImageNet 1k (Deng et al., 2009) dataset for training.

Original: ImageNet1K contains 1,000 classesand the number of training and validation images are 1.28 millions and 50,000, respectively.
Revised: ImageNet1K contains 1,000 classes and the number of training and validation images are 1.28 millions and 50,000, respectively.

Original: Wevalidate the effectiveness of our models in the different datasets proposed in the Brain-Score (Schrimpfet al., 2020) competition.
Revised: We validate the effectiveness of our models in the different datasets proposed in the Brain-Score (Schrimpf et al., 2020a) competition.

Revised: A.2 Custom Scheduler

Original: [Figure]
Revised: [Figure]

Original: The proposed learning rate scheduler is basedon Jeddi et al.
Revised: The proposed learning rate scheduler is based on Jeddi et al.

Original: and is formulated as
Revised: and is formulated as

Original: LR = 0 . 

Revised: LR = 0 .

Original: × e − 0 .
Revised: × e − 0 .

Original: for e = 1 and
Revised: for e = 1 and

Original: LR = 0 . 

Revised: LR = 0 .

Original: e −for 1 < e < = 6 .
Revised: e −for 1 < e < = 6 .

Original: As shownin Figure 2, we start with a small learning rateand then it is smoothly increased for one epoch.
Revised: As shown in Figure 2, we start with a small learning rate and then it is smoothly increased for one epoch.

Original: We empirically found that fine-tuning the transformer for more than 1 epoch resulted in anunder-fitting behavior of the adversarial robustness.
Revised: We empirically found that fine-tuning the transformer for more than 1 epoch resulted in an under-fitting behavior of the adversarial robustness.

Original: After this first epoch, the learning rateis reduced very fast so that model performanceconverges to a steady state, without having toomuch time to overfit on the training data.
Revised: After this first epoch, the learning rate is reduced very fast so that model performance converges to a steady state, without having too much time to overfit on the training data.

Original: A.3 Training Setup
Revised: A.3 Training Setup

Original: We used a pretrained CrossViT-18 † (Chen et al., 2021) downloaded from the timm library that isadversarially trained via a fast gradient sign method (FGSM) attack and random initialization (Wonget al., 2020).
Revised: We used a pretrained CrossViT-18 † (Chen et al., 2021) downloaded from the timm library that is adversarially trained via a fast gradient sign method (FGSM) attack and random initialization (Wong et al., 2020).

Original: We opted for this strategy, known as "Fast Adversarial Training" as it allows a fasteriteration in comparison with other common approaches ( e.g. adversarial training with the PGDattack).
Revised: We opted for this strategy, known as "Fast Adversarial Training" as it allows a faster iteration in comparison with other common approaches ( e.g. adversarial training with the PGD attack).

Original: In particular, all experiments used ϵ = 2 / 255 and step size α = 1 .
Revised: In particular, all experiments used ϵ = 2 / 255 and step size α = 1 .

Original: 25 ϵ as proposed originallyin (Wong et al., 2020).
Revised: 25 ϵ as proposed originally in (Wong et al., 2020).

Original: However, in contrast to the previous method, we follow a 5 epoch fine-tuningapproach with a custom learning rate scheduler in order to avoid underfitting.
Revised: However, in contrast to the previous method, we follow a 5 epoch fine-tuning approach with a custom learning rate scheduler in order to avoid underfitting.

Original: We optimize ournetworks with Adaptive Moment Estimation (Adam a la Kingma & Ba (2014)) and employed mixedprecision for faster training.
Revised: We optimize our networks with Adaptive Moment Estimation (Adam a la Kingma & Ba (2014)) and employed mixed precision for faster training.

Original: All input images were pre-processed with resizing to 256 × 256 followedby standard random cropping and horizontal mirroring.
Revised: All input images were pre-processed with resizing to 256 × 256 followed by standard random cropping and horizontal mirroring.

Original: In case of our best performing model (#991),we additionally incorporated a random grayscale transformation ( p = 0 . 25) and a set of hard rotationtransformations of (0°, 90°, 180°, 270°) – implicitly aiding for rotational invariance – due to thecharacteristics of images appearing in the behavioral benchmark of Rajalingham et al.
Revised: In case of our best performing model (#991), we additionally incorporated a random grayscale transformation ( p = 0 . 25) and a set of hard rotation transformations of (0°, 90°, 180°, 270°) – implicitly aiding for rotational invariance – due to the characteristics of images appearing in the behavioral benchmark of Rajalingham et al.

Original: B Targeted Attacks in Figure
Revised: B Targeted Attacks in Figure

Original: [Table]
Revised: [Table]

Original: C Layers Selected for Brain Areas and Behavioral Benchmark
Revised: C Layers Selected for Brain Areas and Behavioral Benchmark

Original: [Figure]
Revised: [Figure]