Original: Bag of Tricks for Training Brain-Like Deep Neural Networks
Revised: Bag of Tricks for Training Brain-Like Deep Neural Networks

Original: Anonymous Author(s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: The human-level performance of artificial neural networks (ANNs) in visual processing has made them a much-used research subject for understanding how thevisual cortex really works.
Revised: The human-level performance of artificial neural networks (ANNs) in visual processing has made them a much-used research subject for understanding how the visual cortex really works.

Original: To assess how well various types of ANNs representregions of the visual cortex, the Brain-Score platform provides several standardizedbenchmarks.
Revised: To assess how well various types of ANNs represent regions of the visual cortex, the Brain-Score platform provides several standardized benchmarks.

Original: These include the measure of explained variance in ventral streamregions V1, V2, V4, IT and the object recognition behavior in primates.
Revised: These include the measure of explained variance in ventral stream regions V1, V2, V4, IT and the object recognition behavior in primates.

Original: The aim of this work is to find a training procedure that maximizes an ANNsaverage score in the Brain-Score benchmark.
Revised: The aim of this work is to find a training procedure that maximizes an ANNs average score in the Brain-Score benchmark.

Original: The proposed pipeline combinesa customized version of CutMix, heavy use of image augmentations, adversarialrobust training, fixing the train-test resolution discrepancy, and weight averaging.
Revised: The proposed pipeline combines a customized version of CutMix, heavy use of image augmentations, adversarial robust training, fixing the train-test resolution discrepancy, and weight averaging.

Original: Due to its widespread use, memory and computational efficiency, and object recognition performance, the EfficientNet-B1 architecture was used prototypically.
Revised: Due to its widespread use, memory and computational efficiency, and object recognition performance, the EfficientNet-B1 architecture was used prototypically.

Original: Theproposed training methods improve the public object recognition behavior metricscore by 9% and the explained V1 variance by 62%.
Revised: The proposed training methods improve the public object recognition behavior metric score by 9% and the explained V1 variance by 62%, resulting in the best performing models in the Brain-Score competition 2022 .

Original: This is a strong indicator thatfinding the right training strategy might be crucial for developing brain-like ANNs.
Revised: This is a strong indicator that finding the right training strategy might be crucial for developing brain-like ANNs.

Original:  Introduction
Revised:  Introduction

Original: Deep artificial neural networks (ANN) have shown remarkable success in the field of object recognition and therefore became a popular subject of investigation in the neuroscience community.
Revised: Deep artificial neural networks (ANN) have shown remarkable success in the field of object recognition and therefore became a popular subject of investigation in the neuroscience community.

Original: It wasfound that the latent features learned by ANNs after training on large image datasets resemble theneural responses in primates ventral visual cortices better than any other types of models [1, 2].
Revised: It was found that the latent features learned by ANNs after training on large image datasets resemble the neural responses in primates ventral visual cortices better than any other types of models [1, 2].

Original: There is a large number of papers reporting how well ANNs account for various parts of the ventralstream such as V1, V2, V4, inferior temporal (IT) cortex or the object recognition behavior [1, 3, 4,5, 6, 7].
Revised: There is a large number of papers reporting how well ANNs account for various parts of the ventral stream such as V1, V2, V4, inferior temporal (IT) cortex or the object recognition behavior [1, 3, 4, 5, 6, 7].

Original: However, many of these findings can hardly be compared across ANN models or differenttypes of evaluation benchmarks.
Revised: However, many of these findings can hardly be compared across ANN models or different types of evaluation benchmarks.

Original: The Brain-Score platform provides a standardized framework forthese kinds of benchmarks, including an API to (currently) perform 33 neuronal and behavioral 
Revised: The Brain-Score platform provides a standardized framework for these kinds of benchmarks, including an API to (currently) perform 33 neuronal and behavioral brain-similarity tests on ANNs [8].

Original: Many of the approaches in literature for finding brain-like ANNs focus on the search for beneficial
Revised: Many of the approaches in literature for finding brain-like ANNs focus on the search for beneficial

Original: ANN architectures rather than finding an optimal training and data augmentation strategy [6, 7, 9, 10].
Revised: ANN architectures rather than finding an optimal training and data augmentation strategy [6, 7, 9, 10].

Original: This work is aiming towards finding a training procedure for established ANNs in order to maximizetheir average score in the Brain-Score benchmark.
Revised: This work is aiming towards finding a training procedure for established ANNs in order to maximize their average score in the Brain-Score benchmark.

Original: The focus of this work has been on training an
Revised: The focus of this work has been on training an

Original: ANN performing well on the V1 alignment benchmark by Freeman et al.
Revised: ANN performing well on the V1 alignment benchmark by Freeman et al.

Original: [11] and the behavioralalignment benchmark by Rajalingham et al.
Revised: [11] and the behavioral alignment benchmark by Rajalingham et al.

Original: Competitive results in the other evaluation categories
Revised: Competitive results in the other evaluation categories

Original: V2, V4 and IT fell off with it.
Revised: V2, V4 and IT fell off with it.

Original: 1.1 Related Work
Revised: 1.1 Related Work

Original: The positive correlation between adversarial robustness and the ANNs capability to explain brain
Revised: The positive correlation between adversarial robustness and the ANNs capability to explain brain

Original: V1 variance has been demonstrated in literature [6, 12].
Revised: V1 variance has been demonstrated in literature [6, 12].

Original: Dapello et.
Revised: Dapello et.

Original: al. show that ANNs trainedon adversarial perturbed images using a projected gradient descent (PGD) increase the ANNs V1explained variance.
Revised: al. show that ANNs trained on adversarial perturbed images using a projected gradient descent (PGD) increase the ANNs V explained variance.

Original: Though the authors suggest a dedicated architectural backbone for increasing
Revised: Though the authors suggest a dedicated architectural backbone for increasing

Original: V1 explained variance and adversarial robustness of ANNs, the state-of-the-art method for trainingrobust ANNs remains training on adversarial perturbed images [6, 13].
Revised: V1 explained variance and adversarial robustness of ANNs, the state-of-the-art method for training robust ANNs remains training on adversarial perturbed images [6, 13].

Original: The behavior benchmark used in Brain-Score is established by Rajalingham et.
Revised: The behavior benchmark used in Brain-Score is established by Rajalingham et.

Original: al.
Revised: al.

Original: and measuresthe ANNs similarity to primates in recognizing objects in different scenarios [5].
Revised: and measures the ANNs similarity to primates in recognizing objects in different scenarios [5].

Original: For determiningthe object recognition behavior, the test subjects are shown images of 3D object models in variousposes on different backgrounds.
Revised: For determining the object recognition behavior, the test subjects are shown images of 3D object models in various poses on different backgrounds.

Original: Subsequently, the subjects are shown two possible objects (onepreviously shown and a distractor object) and asked to decide which one was shown in the previouscomposition.
Revised: Subsequently, the subjects are shown two possible objects (one previously shown and a distractor object) and asked to decide which one was shown in the previous composition.

Original: The benchmark then measures the decision-similarity between the ANN and the testsubjects.
Revised: The benchmark then measures the decision-similarity between the ANN and the test subjects.

Original: To improve an ANNs behavior-similarity, a first intuitive approach would be to reduce thedata distribution gap between an ANNs training data and the behavior test data.
Revised: To improve an ANNs behavior-similarity, a first intuitive approach would be to reduce the data distribution gap between an ANNs training data and the behavior test data.

Original: The use of image augmentations in training ANNs, improves their generalization ability and thusprevent over-fitting the training data [14].
Revised: The use of image augmentations in training ANNs, improves their generalization ability and thus prevent over-fitting the training data [14].

Original: Augmentation techniques used in the proposed trainingpipeline include geometric transformations, color space augmentations, random erasing and addingnoise.
Revised: Augmentation techniques used in the proposed training pipeline include geometric transformations, color space augmentations, random erasing and adding noise.

Original: For many ImageNet-pretrained vision ANNs, the current best training practice includes randomcropping and resizing at training time, while simply extracting a centered square (center cropping)at test time [15].
Revised: For many ImageNet-pretrained vision ANNs, the current best training practice includes random cropping and resizing at training time, while simply extracting a centered square (center cropping) at test time [15].

Original: This leads to different object sizes at train and test time, resulting in potentiallywasted object recognition performance due to a distribution shift.
Revised: This leads to different object sizes at train and test time, resulting in potentially wasted object recognition performance due to a distribution shift.

Original: The problem can be by increasingthe resizing and cropping resolutions at test time.
Revised: The problem can be by increasing the resizing and cropping resolutions at test time.

Original: Weight averaging (WA) is a well-known generalization technique in deep learning.
Revised: Weight averaging (WA) is a well-known generalization technique in deep learning.

Original: In its simplestform, averaging the learned weights of two or more architectural identical ANNs increases theresulting ANN’s performance.
Revised: In its simplest form, averaging the learned weights of two or more architectural identical ANNs increases the resulting ANN’s performance.

Original: It also leads to wider optima than those found by stochastic gradientdescent during training [16].
Revised: It also leads to wider optima than those found by stochastic gradient descent during training [16].

Original:  Methods and Results
Revised:  Methods and Results

Original: The above-mentioned methods combine different concepts to improve an ANNs Brain-Score benchmarks.
Revised: The above-mentioned methods combine different concepts to improve an ANNs Brain-Score benchmarks.

Original: They represent a mix of regularization, generalization, data distribution shifts and adversarialrobustness.
Revised: They represent a mix of regularization, generalization, data distribution shifts and adversarial robustness.

Original: Despite V1 alignment and adversarial robustness as well as behavioral alignment andclassification accuracy generally don’t correlate very strong, the robustness and accuracy gains fromsome techniques do correlate with the corresponding V1 and behavior metrics.
Revised: Despite V1 alignment and adversarial robustness as well as behavioral alignment and classification accuracy generally don’t correlate very strong, the robustness and accuracy gains from some techniques do correlate with the corresponding V1 and behavior metrics.

Original: Fixing the train-testresolution discrepancy (FixRes) of EfficientNet-B1 for example changes an ANNs Brain-Scorebehavior metric in the same cyclic pattern as observed for the object classification accuracy (see
Revised: Fixing the train-test resolution discrepancy (FixRes) of EfficientNet-B1 for example changes an ANNs Brain-Score behavior metric in the same cyclic pattern as observed for the object classification accuracy (see

Original: Figure 1 and [17]).
Revised: Figure 1 and [17]).

Original: Also, the behavior score gains from weight averaging are comparable to theimage classification accuracy gains from this method.
Revised: Also, the behavior score gains from weight averaging are comparable to the image classification accuracy gains from this method.

Original: To evaluate the benefit of the proposed trainingand evaluation pipeline, an ablation study was conducted.
Revised: To evaluate the benefit of the proposed training and evaluation pipeline, an ablation study was conducted.

Original: The study is split into behavior metricablation and V1 metric ablation, both using different techniques.
Revised: The study is split into behavior metric ablation and V1 metric ablation, both using different techniques.

Original: An intuitive approach to train ANNs to be not distracted by the background image presented in thebehavioral benchmark [5] would be a adjusted form of the CutMix augmentation [18].
Revised: An intuitive approach to train ANNs to be not distracted by the background image presented in the behavioral benchmark [5] would be a adjusted form of the CutMix augmentation [18].

Original: In contrast tothe original CutMix, the proposed method only accounts for the pasted image to be the correct class(Table 3, "Foreground-CutMix").
Revised: In contrast to the original CutMix, the proposed method only accounts for the pasted image to be the correct class (Table 3, "Foreground-CutMix").

Original: This technique will shift the training data towards the data shownin the behavioral benchmark.
Revised: This technique will shift the training data towards the data shown in the behavioral benchmark.

Original: The augmentations used in the ablation study (referred to as "heavy augmentations") include imageshifting, rotation, elastic deformation, brightness and contrast adjustments, motion blurring, cutout,adding Gaussian and ISO noise [19].
Revised: The augmentations used in the ablation study (referred to as "heavy augmentations") include image shifting, rotation, elastic deformation, brightness and contrast adjustments, motion blurring, cutout, adding Gaussian and ISO noise [19].

Original: The adversarial robust training used for improving the V1 metric adversely effects the ANNs objectclassification performance, and thus the behavior score.
Revised: The adversarial robust training used for improving the V1 metric adversely effects the ANNs object classification performance, and thus the behavior score.

Original: To overcome this, the final ANN was firsttrained adversarial robust under the optimal parameters (see Table 2).
Revised: To overcome this, the final ANN was first trained adversarial robust under the optimal parameters (see Table 2).

Original: Subsequently the first half ofthe ANNs layers were frozen (including Batch Norm statistics), and it was fine-tuned for behaviorresponse using the proposed techniques in 1.
Revised: Subsequently the first half of the ANNs layers were frozen (including Batch Norm statistics), and it was fine-tuned for behavior response using the proposed techniques in 1.

Original: The layers that explain most of V1 variance are likelyto be found among the first half of all layers.
Revised: The layers that explain most of V1 variance are likely to be found among the first half of all layers.

Original: The behavioral response however is read out in thepenultimate layer.
Revised: The behavioral response however is read out in the penultimate layer.

Original: The presented ablation studies were conducted without any layer freezing.
Revised: The presented ablation studies were conducted without any layer freezing.

Original: [Table]
Revised: [Table]

Original: The frozen layer fine-tuning for behavioral response leads to a lower behavior score compared toall-layer training due to the limited capacity.
Revised: The frozen layer fine-tuning for behavioral response leads to a lower behavior score compared to all-layer training due to the limited capacity.

Original: However, the proposed training techniques boost thepublic behavior score from 0.4527 ( L ∞ , ϵ = 8 / 255 adversarially trained) to 0.5271 ( L ∞ , ϵ = 8 / 255adversarially trained and behavior fine-tuned) while keeping the optimized V1 explained variance.
Revised: However, the proposed training techniques boost the public behavior score from 0.4527 ( L ∞ , ϵ = 8 / 255 adversarially trained) to 0.5271 ( L ∞ , ϵ = 8 /adversarially trained and behavior fine-tuned) while keeping the optimized V1 explained variance.

Original: The proposed methods were validated on a ResNet50 [20] to demonstrate their generalizability.
Revised: The proposed methods were validated on a ResNet50 [20] to demonstrate their generalizability.

Original: Similar improvements in explained V1 variance and behavioral response can be achieved comparedto the studied EfficientNet-B1 architecture.
Revised: Similar improvements in explained V1 variance and behavioral response can be achieved compared to the studied EfficientNet-B1 architecture.

Original: The ANN validated without robust training is the best performing behavior ANN (see Table 1).
Revised: The ANN validated without robust training is the best performing behavior ANN (see Table 1).

Original: [Table]
Revised: [Table]

Original: 2013.V1-pls) w/o robust training 0. L2 , ϵ = 3 .0. L ∞ , ϵ = 8 /0.
Revised: L

Original: L ∞ , ϵ = 4 /0.
Revised: [Table]

Original:  Discussion
Revised:  Discussion

Original: The combination of adversarial robust training, layer freezing and behavioral metric aware datadistribution shifting leads to an improved explained V1 variance and a competitive behavioral metricresponse.
Revised: The combination of adversarial robust training, layer freezing and behavioral metric aware data distribution shifting leads to an improved explained V1 variance and a competitive behavioral metric response.

Original: It still remains unclear whether the proposed methods really lead towards  more biologicallyplausible ANNs or if they are just ways to exploit the Brain-Score benchmark.
Revised: Despite the proposed methods lead to a winning model in the Brain-Score competition 2022 and present a new state-of-the-art performance in the Brain-Score benchmark overall, it still remains unclear whether they lead towards biologically more plausible ANNs or if they are just ways to exploit the Brain-Score benchmark.

Original: However, the furtherdevelopment of brain-like ANNs should address the influence of an optimized training pipeline as itmight be a way to push a biologically plausible ANN towards its frontiers of explaining primatesvisual cortex responses.
Revised: However, the further development of brain-like ANNs should address the influence of an optimized training pipeline as it might be a way to push a biologically plausible ANN towards its frontiers of explaining primates visual cortex responses

Original: A Appendix
Revised: A Appendix

Original: As the images presented in the Rajalingham behavior benchmark are composed of an 3D objects randomly171 pasted on different backgrounds, it seemed advantageous to simulate this type of input.
Revised: As the images presented in the Rajalingham behavior benchmark are composed of an 3D objects randomly pasted on different backgrounds, it seemed advantageous to simulate this type of input.

Original: An intuitive approach issimply pasting parts of ImageNet images on top of a background image and assign the pasted image label to the173 whole image.
Revised: An intuitive approach is simply pasting parts of ImageNet images on top of a background image and assign the pasted image label to the whole image.

Original: This method is called Foreground-CutMix in the present work.
Revised: This method is called Foreground-CutMix in the present work.

Original: This induces the  that an  be  and shifts 
Revised: This induces the concept that an object can be anywhere in the image and shifts away from photogra

Original: [Table]
Revised: [Table]

Original: The train-test resolution discrepancy of ANNs that were trained using random resizing and cropping and tested175 using center cropping leads to a decrease in classification accuracy.
Revised: The train-test resolution discrepancy of ANNs that were trained using random resizing and cropping and tested using center cropping leads to a decrease in classification accuracy.

Original: This can be overcome by manipulating thefinal average pooling layer or by simply finding the best performing test resolution.
Revised: This can be overcome by manipulating the final average pooling layer or by simply finding the best performing test resolution.

Original: For an EfficientNet-B1177 trained on images that are resized to 272x272 pixels and cropped to 240x240 pixels, the behavior-benchmarkoptimal resolution at test time was empirically determined to be 324x324 pixels.
Revised: For an EfficientNet-B1 trained on images that are resized to 272x272 pixels and cropped to 240x240 pixels, the behavior-benchmarkoptimal resolution at test time was empirically determined to be 324x324 pixels.

Original: A similar effect is observed for179 the ImageNet object classification accuracy in [15], though the optimal EfficientNet-B1 resolution is found to be384x384 pixels for object classification.[Figure] [Table] [Figure]
Revised: A similar effect is observed for the ImageNet object classification accuracy in [15], though the optimal EfficientNet-B1 resolution is found to be 384x384 pixels for object classification.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]