Original: S YSTEM IDENTIFICATION OF NEURAL SYSTEMS : I F WE GOT IT RIGHT , WOULD WE KNOW ?
Revised: S YSTEM IDENTIFICATION OF NEURAL SYSTEMS : I F WE GOT IT RIGHT , WOULD WE KNOW ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Various artificial neural networks developed by engineers are now proposed as models of parts of the brain, such as the ventral stream in the primate visual cortex.
Revised: Various artificial neural networks developed by engineers are now proposed as models of parts of the brain, such as the ventral stream in the primate visual cortex.

Original: The network activations are compared to recordings of biological neurons, and good performance in reproducing neural responses is considered to support the model’s validity.
Revised: The network activations are compared to recordings of biological neurons, and good performance in reproducing neural responses is considered to support the model’s validity.

Original: This system identification approach, however, is only part of the traditional ways to develop and test models in the natural sciences.
Revised: This system identification approach, however, is only part of the traditional ways to develop and test models in the natural sciences.

Original: A key question is how much the ability to predict neural responses tells us.
Revised: A key question is how much the ability to predict neural responses tells us.

Original: In particular, do these functional tests about neuron activation allow us to distinguish between different model architectures?
Revised: In particular, do these functional tests about neuron activation allow us to distinguish between different model architectures?

Original: We benchmark existing techniques to correctly identify a model by replacing brain recordings with known ground truth models.
Revised: We benchmark existing techniques to correctly identify a model by replacing brain recordings with known ground truth models.

Original: We evaluate the most commonly used identification approaches, such as a linear encoding model and centered kernel alignment.
Revised: We evaluate the most commonly used identification approaches, such as a linear encoding model and centered kernel alignment.

Original: Even in the setting where the correct model is among the candidates, system identification performance is quite variable; it also depends significantly on factors independent of the ground truth architecture, such as stimuli images.
Revised: Even in the setting where the correct model is among the candidates, system identification performance is quite variable; it also depends significantly on factors independent of the ground truth architecture, such as stimuli images.

Original: In addition, we show the limitations of using functional similarity scores in identifying higher-level architectural motifs.
Revised: In addition, we show the limitations of using functional similarity scores in identifying higher-level architectural motifs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Over the last two decades, the dominant approach for machine learning engineers in search of better performance has been to use standard benchmarks to rank networks from most relevant to least relevant.
Revised: Over the last two decades, the dominant approach for machine learning engineers in search of better performance has been to use standard benchmarks to rank networks from most relevant to least relevant.

Original: This practice has driven much of the progress in the machine learning community.
Revised: This practice has driven much of the progress in the machine learning community.

Original: A standard comparison benchmark enables the broad validation of successful ideas.
Revised: A standard comparison benchmark enables the broad validation of successful ideas.

Original: Recently such benchmarks have found their way into neuroscience with the advent of experimental frameworks like Brain-Score (Schrimpf et al., 2020) and Algonauts (Cichy et al., 2021), where artificial models compete to predict recordings from real neurons in animal brains.
Revised: Recently such benchmarks have found their way into neuroscience with the advent of experimental frameworks like Brain-Score (Schrimpf et al., 2020) and Algonauts (Cichy et al., 2021), where artificial models compete to predict recordings from real neurons in animal brains.

Original: Can engineering approaches like this be helpful in the natural sciences?
Revised: Can engineering approaches like this be helpful in the natural sciences?

Original: The answer is clearly yes: the “engineering approach” described above ranks models that predict neural responses better as better models of animal brains.
Revised: The answer is clearly yes: the “engineering approach” described above ranks models that predict neural responses better as better models of animal brains.

Original: While such rankings may be a good measure of absolute performance in approximating the neural responses,  it is an open question whether they are sufficient.
Revised: While such rankings may be a good measure of absolute performance in approximating the neural responses, which on its own is valuable for various applications (Bashivan et al., 2019), it is an open question whether they are sufficient.

Original: In neuroscience, understanding natural intelligence at the level of the underlying neural circuits requires developing model systems that reproduce the abilities of their biological analogs while respecting the constraints provided by biology, including anatomy and biophysics (Marr & Poggio, 1976; Schaeffer et al., 2022).
Revised: In neuroscience, understanding natural intelligence at the level of the underlying neural circuits requires developing model systems that reproduce the abilities of their biological analogs while respecting the constraints provided by biology, including anatomy and biophysics (Marr & Poggio, 1976; Schaeffer et al., 2022).

Original: A model that reproduces neural responses well but turns out to require connectivity or biophysical mechanisms that are different from the biological ones is thereby falsified.
Revised: A model that reproduces neural responses well but turns out to require connectivity or biophysical mechanisms that are different from the biological ones is thereby falsified.

Original: Consider the conjecture that the similarity of responses between model units and brain neurons allows us to conclude that brain activity fits better, for instance, a convolutional motif rather than a dense architecture.
Revised: Consider the conjecture that the similarity of responses between model units and brain neurons allows us to conclude that brain activity fits better, for instance, a convolutional motif rather than a dense architecture.

Original: If this were true, it would mean that functional similarity over large data sets effectively constrains architecture.
Revised: If this were true, it would mean that functional similarity over large data sets effectively constrains architecture.

Original: Then the need for a separate test of the model at the level of anatomy would become, at least in part, less critical for model validation.
Revised: Then the need for a separate test of the model at the level of anatomy would become, at least in part, less critical for model validation.

Original: Therefore, we ask the question: could functional similarity be a reliable predictor of architectural similarity?
Revised: Therefore, we ask the question: could functional similarity be a reliable predictor of architectural similarity?

Original: We describe an attempt to benchmark the most popular similarity techniques by replacing the brain recordings with data generated by various known networks with drastically different architectural motifs, such as convolution vs. attention.
Revised: We describe an attempt to benchmark the most popular similarity techniques by replacing the brain recordings with data generated by various known networks with drastically different architectural motifs, such as convolution vs. attention.

Original: Such a setting provides a valuable upper bound to the identifiability of anatomical differences.
Revised: Such a setting provides a valuable upper bound to the identifiability of anatomical differences.

Original: 1. S YSTEM IDENTIFICATION FROM LEADERBOARDS
Revised: 1. S YSTEM IDENTIFICATION FROM LEADERBOARDS

Original: When artificial models are compared against common biological benchmarks for predictivity (Yamins & DiCarlo, 2016), models with the top score are deemed better models for neuroscience.
Revised: When artificial models are compared against common biological benchmarks for predictivity (Yamins & DiCarlo, 2016), models with the top score are deemed better models for neuroscience.

Original: As improvements to scores are made over time, ideally, more relevant candidates emerge.
Revised: As improvements to scores are made over time, ideally, more relevant candidates emerge.

Original: Nevertheless, if two artificial models with distinctly different architectures, trained on the same data, happen to be similar in reproducing neural activities (target model), then it would be impossible to conclude what accounts for the similarity.
Revised: Nevertheless, if two artificial models with distinctly different architectures, trained on the same data, happen to be similar in reproducing neural activities (target model), then it would be impossible to conclude what accounts for the similarity.

Original: It can be biologically relevant motifs from each architecture , the properties of the stimulus input , or similarity metric .
Revised: It can be biologically relevant motifs from each architecture , the properties of the stimulus input , or similarity metric .

Original: Such ambiguity is due to the many-to-one mapping of a model onto a leaderboard score.
Revised: Such ambiguity is due to the many-to-one mapping of a model onto a leaderboard score.

Original: Our work shows that multiple factors play a role in representational similarities.
Revised: Our work shows that multiple factors play a role in representational similarities.

Original: An interesting example is offered by Chang et al.
Revised: An interesting example is offered by Chang et al.

Original: (2021), which compares many different models with respect to their ability to reproduce neural responses in IT to face images and concludes that the 2D morphable model is best.
Revised: (2021), which compares many different models with respect to their ability to reproduce neural responses in IT to face images and concludes that the 2D morphable model is best.

Original: Operations required in the specific model, such as correspondece and vectorization do not have an apparent biological implementation in terms of neurons and synapses.
Revised: Operations required in the specific model, such as correspondece and vectorization do not have an apparent biological implementation in terms of neurons and synapses.

Original: Nonetheless, it is perhaps not too surprising that the model can predict IT responses well, as there are multiple confounds besides the biological constraints, which affect the neural predictivity.
Revised: Nonetheless, it is perhaps not too surprising that the model can predict IT responses well, as there are multiple confounds besides the biological constraints, which affect the neural predictivity.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: While the analogy between neural network models and the brain has been well validated (Bashivan et al., 2019), the extent of this correspondence across multiple levels (Marr & Poggio, 1976) has been taken for granted.
Revised: While the analogy between neural network models and the brain has been well validated (Bashivan et al., 2019), the extent of this correspondence across multiple levels (Marr & Poggio, 1976) has been taken for granted.

Original: This assumed correspondence could be attributed to methodological limitations of evaluating such models simultaneously across all levels.
Revised: This assumed correspondence could be attributed to methodological limitations of evaluating such models simultaneously across all levels.

Original: Jonas & Kording (2017) investigated the robustness of standard analysis techniques in neuroscience with a microprocessor as a ground-truth model to determine the boundaries of what conclusions could be drawn about a known system.
Revised: Jonas & Kording (2017) investigated the robustness of standard analysis techniques in neuroscience with a microprocessor as a ground-truth model to determine the boundaries of what conclusions could be drawn about a known system.

Original: The presumption of correspondence could also be attributed to underappreciated variability from model hyperparameters (Schaeffer et al., 2022).
Revised: The presumption of correspondence could also be attributed to underappreciated variability from model hyperparameters (Schaeffer et al., 2022).

Original: In a similar spirit to Jonas & Kording (2017); Lazebnik (2002), we evaluate system identification on a known ground-truth model to establish the boundaries of what architectural motifs can be reliably uncovered.
Revised: In a similar spirit to Jonas & Kording (2017); Lazebnik (2002), we evaluate system identification on a known ground-truth model to establish the boundaries of what architectural motifs can be reliably uncovered.

Original: We perform our analysis under favorable experimental conditions to establish an upper bound.
Revised: We perform our analysis under favorable experimental conditions to establish an upper bound.

Original: As modern neural network models have grown more prominent in unison with the corresponding resources to train these models, pre-trained reference models have become more widely available in research (Wightman, 2019).
Revised: As modern neural network models have grown more prominent in unison with the corresponding resources to train these models, pre-trained reference models have become more widely available in research (Wightman, 2019).

Original: Consequently, the need to compare these references along different metrics has followed suit.
Revised: Consequently, the need to compare these references along different metrics has followed suit.

Original: Kornblith et al.
Revised: Kornblith et al.

Original: (2019) explored using different similarity measures between the layers of artificial neural network models.
Revised: (2019) explored using different similarity measures between the layers of artificial neural network models.

Original: Kornblith et al.
Revised: Kornblith et al.

Original: (2019) propose various properties a similarity measure should be invariant such as orthogonal transformations and isotropic scaling while not invariant to invertible linear transformations.
Revised: (2019) propose various properties a similarity measure should be invariant such as orthogonal transformations and isotropic scaling while not invariant to invertible linear transformations.

Original: Kornblith et al.
Revised: Kornblith et al.

Original: (2019) found centered kernel alignment (CKA), a method very similar to Representation Similarity Analysis (Kriegeskorte et al., 2008), to best satisfy these requirements.
Revised: (2019) found centered kernel alignment (CKA), a method very similar to Representation Similarity Analysis (Kriegeskorte et al., 2008), to best satisfy these requirements.

Original: Ding et al.
Revised: Ding et al.

Original: (2021) explored the sensitivity of methods like canonical correlation analysis, CKA, and orthogonal procrustes distance to changes in factors that do not impact the functional behavior of neural network models.
Revised: (2021) explored the sensitivity of methods like canonical correlation analysis, CKA, and orthogonal procrustes distance to changes in factors that do not impact the functional behavior of neural network models.

Original:  B ACKGROUND AND M ETHODS
Revised:  B ACKGROUND AND M ETHODS

Original: The two predominant approaches to evaluating computational models of the brain are using metrics based on linear encoding analysis for neural predictivity and population-level representation similarity.
Revised: The two predominant approaches to evaluating computational models of the brain are using metrics based on linear encoding analysis for neural predictivity and population-level representation similarity.

Original: The first measures how well a model can predict the activations of individual units, whereas the second metric measures how correlated the variance of internal representations is.
Revised: The first measures how well a model can predict the activations of individual units, whereas the second metric measures how correlated the variance of internal representations is.

Original: We study the following neural predictivity scores consistent with the typical approaches: Linear Regression and
Revised: We study the following neural predictivity scores consistent with the typical approaches: Linear Regression and

Original: Centered Kernel Alignment (CKA).
Revised: Centered Kernel Alignment (CKA).

Original: In computational neuroscience, we usually have a neural system (brain) that we are interested in modeling.
Revised: In computational neuroscience, we usually have a neural system (brain) that we are interested in modeling.

Original: We call this network a target and the proposed candidate model a source .
Revised: We call this network a target and the proposed candidate model a source .

Original: Formally, for a layer with p 1 units in a source model, let X ∈ R n × p 1 be the matrix of representations with pfeatures over n stimulus images.
Revised: Formally, for a layer with p 1 units in a source model, let X ∈ R n × p 1 be the matrix of representations with p 1 features over n stimulus images.

Original: Similarly, let Y ∈ R n × p 2 be a matrix of representations with pfeatures of the target model (or layer) on the same n stimulus images.
Revised: Similarly, let Y ∈ R n × p 2 be a matrix of representations with p 2 features of the target model (or layer) on the same n stimulus images.

Original: Unless otherwise noted, we subsample 3000 target units to test an analogous condition as in biology, where recordings are far from exhaustive.
Revised: Unless otherwise noted, we subsample 3000 target units to test an analogous condition as in biology, where recordings are far from exhaustive.

Revised: Our analyses are partially dependent upon the target coverage, and we later examine the effect of increasing the number of target units.

Original: 3. E NCODING M ODEL : L INEAR R EGRESSION
Revised: 3. E NCODING M ODEL : L INEAR R EGRESSION

Original: Following the procedure developed by previous works (Schrimpf et al., 2020; Yamins et al., 2014;
Revised: Following the procedure developed by previous works (Schrimpf et al., 2020; Yamins et al., 2014;

Original: Conwell et al., 2021), we linearly project the feature space of a single layer in a source model to map onto a single unit in a target model (a column of Y ).
Revised: Conwell et al., 2021; Kar et al., 2019; Mitchell et al., 2008), we linearly project the feature space of a single layer in a source model to map onto a single unit in a target model (a column of Y ).

Original: The linear regression score is the Pearson’s correlation r ( · , · ) coefficient between the predicted responses of a source model and the ground-truth target responses to a set of stimulus images.
Revised: The linear regression score is the Pearson’s correlation r ( · , · ) coefficient between the predicted responses of a source model and the ground-truth target responses to a set of stimulus images.

Original: We use the regularization parameter λ = 1 for our main experiments, and we show the effect of varying the value in the Appendix.

Original: [Equation]
Revised: [Equation]

Original: We first extract activations on the same set of stimulus images for source and target models.
Revised: We first extract activations on the same set of stimulus images for source and target models.

Original: To reduce computational costs without sacrificing predictivity, we apply sparse random projection S ∈ R p 1 × q 1 for q 1 << p 1 , on the activations of the source model (Conwell et al., 2021).
Revised: To reduce computational costs without sacrificing predictivity, we apply sparse random projection S ∈ R p 1 × q 1 for q 1 << p 1 , on the activations of the source model (Conwell et al., 2021).

Original: This projection reduces the dimensionality of the features to q 1 while still preserving relative distances between points (Li et al., 2006).
Revised: This projection reduces the dimensionality of the features to q 1 while still preserving relative distances between points (Li et al., 2006).

Original: Unlike principle component analysis, sparse random projection is a datasetindependent dimensionality reduction method.
Revised: Unlike principal component analysis, sparse random projection is a datasetindependent dimensionality reduction method.

Original: This removes any data-dependent confounds from our processing pipeline for linear regression and isolates dataset dependence into our variables of interest: linear regression and candidate model.
Revised: This removes any data-dependent confounds from our processing pipeline for linear regression and isolates dataset dependence into our variables of interest: linear regression and candidate model.

Original: We expect computationally scalable techniques like this to become more common as deep learning model candidates continue to become larger and trained on more diverse

Original: We apply ridge regression on every layer of a source model to predict a target unit using these features.
Revised: We apply ridge regression on every layer of a source model to predict a target unit using these features.

Original: We use 90% of the stimulus images for linear fitting and test on 10%, cross-validatedtimes.
Revised: We use nested cross-validations in which the regularization parameter λ is chosen in the inner loop and a linear model is fitted in the outer loop. The list of tested λ is [0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 , 100] . We use 5-fold cross-validation for both inner and outer loops.

Original: As there are multiple target units, the median of Pearson’s correlation coefficients between predicted and true responses is the aggregate score for layer-wise comparison between source and target models.
Revised: As there are multiple target units, the median of Pearson’s correlation coefficients between predicted and true responses is the aggregate score for layer-wise comparison between source and target models.

Original: Note that a layer of a target model is usually assumed to correspond to a visual area, e.g., V1 or IT, in the visual cortex.
Revised: Note that a layer of a target model is usually assumed to correspond to a visual area, e.g., V1 or IT, in the visual cortex.

Original: For a layer-mapped model, we report maximum linear regression scores across source layers for target layers.
Revised: For a layer-mapped model, we report maximum linear regression scores across source layers for target layers.

Original: 3. C ENTERED K ERNEL A LIGNMENT
Revised: 3. C ENTERED K ERNEL A LIGNMENT

Original: Another widely used type of metric builds upon the idea of measuring the representational similarity between the activations of two neural networks for each pair of images.
Revised: Another widely used type of metric builds upon the idea of measuring the representational similarity between the activations of two neural networks for each pair of images.

Original: While variants of this metric abound, including RSA or re-weighted RSA (Kriegeskorte et al., 2008; Khaligh-Razavi et al., 2017), we use CKA (Cortes et al., 2012) as (Kornblith et al., 2019) showed strong correspondence between layers of models trained with different initializations, which we will further discuss as a validity test we perform.
Revised: While variants of this metric abound, including RSA or re-weighted RSA (Kriegeskorte et al., 2008; Khaligh-Razavi et al., 2017), we use CKA (Cortes et al., 2012) as (Kornblith et al., 2019) showed strong correspondence between layers of models trained with different initializations, which we will further discuss as a validity test we perform.

Original: We consider linear CKA in this work:
Revised: We consider linear CKA in this work:

Original: [Equation]
Revised: [Equation]

Original: (Kornblith et al., 2019) showed that the variance explained by (unregularized) linear regression accounts for the singular values of the source representation.
Revised: (Kornblith et al., 2019) showed that the variance explained by (unregularized) linear regression accounts for the singular values of the source representation.

Original: In contrast, linear CKA depends on the singular values of both target and source representations.
Revised: In contrast, linear CKA depends on the singular values of both target and source representations.

Original: Recent work (Diedrichsen et al., 2020) notes that linear CKA is equivalent to a whitened representational dissimilarity matrix (RDM) in
Revised: Recent work (Diedrichsen et al., 2020) notes that linear CKA is equivalent to a whitened representational dissimilarity matrix (RDM) in

Original: RSA under certain conditions.
Revised: RSA under certain conditions.

Original: We also call CKA a neural predictivity score because a target network is observable, whereas a source network gives predicted responses.
Revised: We also call CKA a neural predictivity score because a target network is observable, whereas a source network gives predicted responses.

Original: A distinct feature of linear regression from CKA is that it is parametric with a fitting process.

Original: Fitting the parameters can introduce variance in the predictions (i.e., bias-variance tradeoff), which is cor- rected for using holdout data from a validation set. In contrast, CKA does not have parameters and does not predict a target. However, in a similar fashion to linear regression, the variance of the score CKA produces can be estimated with multiple sets of held-out data.
Revised: [Figure]

Original: This can provide a meaningful measure of confidence in the score and the reliability of identifiability.

Original: In this work, we focus on pure identifiability and leave reliability of identification over datasets as a future direction.

Original: 3. I DENTIFIABILITY I NDEX
Revised: 3. I DENTIFIABILITY I NDEX

Original: To quantify how selective predictivity scores are when a source matches the target architecture compared to when the architecture differs between source and target networks, we define an identifiability index as:
Revised: To quantify how selective predictivity scores are when a source matches the target architecture compared to when the architecture differs between source and target networks, we define an identifiability index as:

Original: [Equation]
Revised: [Equation]

Original: In brief, it is a normalized difference between the score for the true positive and the mean score for the true negatives.
Revised: In brief, it is a normalized difference between the score for the true positive and the mean score for the true negatives.

Original: Previous works (Dobs et al., 2022; Freiwald & Tsao, 2010) defined selectivity indices in the same way in similar contexts, such as the selectivity of a neuron to specific tasks.
Revised: Previous works (Dobs et al., 2022; Freiwald & Tsao, 2010) defined selectivity indices in the same way in similar contexts, such as the selectivity of a neuron to specific tasks.

Original: 3. S IMULATED E NVIRONMENT
Revised: 3. S IMULATED E NVIRONMENT

Original: If a target network is a brain, it is essentially a black box, making it challenging to understand the properties or limitations of the comparison metrics.
Revised: If a target network is a brain, it is essentially a black box, making it challenging to understand the properties or limitations of the comparison metrics.

Original: Therefore, we instead use artificial neural networks of our choice as targets for our experiments.
Revised: Therefore, we instead use artificial neural networks of our choice as targets for our experiments.

Original: We investigate the reliability of a metric to compare models, mainly to discriminate the underlying computations specified by the model’s architecture.
Revised: We investigate the reliability of a metric to compare models, mainly to discriminate the underlying computations specified by the model’s architecture.

Original: We intentionally create favorable conditions for identifiability in a simulated environment where the ground truth model is a candidate among the source models.
Revised: We intentionally create favorable conditions for identifiability in a simulated environment where the ground truth model is a candidate among the source models.

Original: Taking these ideal conditions further, our target and source models are deterministic and do not include adverse conditions typically encountered in biological recordings, such as noise and temporal processing.
Revised: Taking these ideal conditions further, our target and source models are deterministic and do not include adverse conditions typically encountered in biological recordings, such as noise and temporal processing.

Original: We consider the following architectures:
Revised: We consider the following architectures:

Original: Convolutional Networks: AlexNet (Krizhevsky et al., 2012), VGG11 (Simonyan & Zisserman, 2014), ResNet18 (He
Revised: Convolutional Networks : AlexNet (Krizhevsky et al., 2012), VGG11 (Simonyan & Zisserman, 2014), ResNet18 (He

Original: et al., 2016)
Revised: et al., 2016)

Revised: Recurrent Networks: CORnet-S (Kubilius et al., 2019)

Original: Transformer Networks: ViT-B/32 (Dosovitskiy et al., 2020)
Revised: Transformer Networks: ViT-B/32 (Dosovitskiy et al., 2020)

Original: Mixer Networks: MLP-Mixer-B/16 (Tolstikhin et al., 2021)
Revised: Mixer Networks: MLP-Mixer-B/16 (Tolstikhin et al., 2021)

Original: These architectures are emblematic of the vision-based models used today.
Revised: These architectures are emblematic of the vision-based models used today.

Original: Each architecture has a distinct motif, making it unique from other models.
Revised: Each architecture has a distinct motif, making it unique from other models.

Original: For example, transformer networks use the soft-attention operation as a core motif, whereas convolutional networks use convolution.
Revised: For example, transformer networks use the soft-attention operation as a core motif, whereas convolutional networks use convolution.

Revised: Recurrent networks implement feedback connections which may be critical in the visual cortex for object recognition (Kubilius et al., 2019; Kar et al., 2019).

Original: Moreover, mixer networks (Tolstikhin et al., 2021; Touvron et al., 2021; Melas-Kyriazi, 2021) uniquely perform fully-connected operations over image patches, alternating between the feature and patch dimensions. 
Revised: Moreover, mixer networks (Tolstikhin et al., 2021; Touvron et al., 2021; Melas-Kyriazi, 2021) uniquely perform fully-connected operations over image patches, alternating between the feature and patch dimensions. [Figure]

Revised: based on the aggregate median scores (Schrimpf et al., 2020).

Original:  R ESULTS 4. D IFFERENT MODELS TRAINED ON A LARGE - SCALE DATASET REACH EQUIVALENT NEURAL PREDICTIVITY
Revised:  R ESULTS 4. D IFFERENT MODELS TRAINED ON A LARGE - SCALE DATASET REACH EQUIVALENT NEURAL PREDICTIVITY

Original: We compare various artificial neural networks with publicly shared neural recordings in primates (Majaj et al., 2015; Freeman et al., 2013) via the Brain-Score framework (Schrimpf et al., 2020).
Revised: We compare various artificial neural networks with publicly shared neural recordings in primates (Majaj et al., 2015; Freeman et al., 2013) via the Brain-Score framework (Schrimpf et al., 2020).

Original: Our experiments show that the differences between markedly different neural network architectures are minimal after training (Figure 1), consistent with the previous work (Schrimpf et al., 2020; Kubilius et al., 2019; Conwell et al., 2021).
Revised: Our experiments show that the differences between markedly different neural network architectures are minimal after training (Figure 1), consistent with the previous work (Schrimpf et al., 2020; Kubilius et al., 2019; Conwell et al., 2021).

Original: Previous works focused on the relative ranking of models and investigated which model yields the highest score.
Revised: Previous works focused on the relative ranking of models and investigated which model yields the highest score.

Original: However, if we take a closer look at the result, the performance difference is minimal, with the range of scores having a standard deviation < 0 .(for V2=0.021, V4=0.023, IT=0.016) except for V1.
Revised: However, if we take a closer look at the result, the performance difference is minimal, with the range of scores having a standard deviation < 0 .(for V2=0.021, V4=0.023, IT=0.016) except for V1.

Original: For V1, VOneNets (Dapello et al., 2020), which explicitly build in properties observed from experimental works in neuroscience, significantly outperform other models.Notably, the models we consider have quite different architectures based [Figure] [Figure]
Revised: For V1, VOneNets (Dapello et al., 2020), which explicitly build in properties observed from experimental works in neuroscience, significantly outperform other models.

Original: 8 Layer Number 0.

Original: Target: VGG

Original: AlexNet

Original: VGG

Original: ResNet

Original: ViT-B/

Original: MLP-Mixer-B/     8 Layer Number 0.

Original: Target: ResNet 0 2 4 6 8 10 Layer Number 0.

Original: Target: ViT-B/ 0 4 8 12 16 20 Layer Number 0.

Original: [Figure]

Original: : MLP

Original: [Figure] -B/ [Figure] on combinations of various components, such as convolutional layers, attention layers, and skip connections.
Revised: Notably, the models we consider have quite different architectures based on combinations of various components, such as convolutional layers, attention layers, and skip connections.

Original: This suggests that architectures with different computational operations reach almost equivalent performance after training on the same large-scale dataset, i e., ImageNet.
Revised: This suggests that architectures with different computational operations reach almost equivalent performance after training on the same large-scale dataset, i.e., ImageNet.

Original: 4. I DENTIFICATION OF ARCHITECTURES IN AN IDEAL SETTING
Revised: 4. I DENTIFICATION OF ARCHITECTURES IN AN IDEAL SETTING

Original: One potential interpretation of the result would be that different neural network architectures are equally good (or bad) models of the visual cortex.
Revised: One potential interpretation of the result would be that different neural network architectures are equally good (or bad) models of the visual cortex.

Original: An alternative explanation would be that the method we use to compare models with the brain has limitations in identifying precise computational operations.
Revised: An alternative explanation would be that the method we use to compare models with the brain has limitations in identifying precise computational operations.

Original: To test the hypothesis, we focus on where underlying target neural networks are known instead of being a black box, as with biological brains.
Revised: To test the hypothesis, we focus on where underlying target neural networks are known instead of being a black box, as with biological brains.

Original: Specifically, we replace the target neural [Figure] source and target networks.
Revised: Specifically, we replace the target neural recordings with artificial neural network activations.

Original: The experimental setup is identical to

Original: Figure 2 besides using CKA instead of linear regression as the metric.

Original: As in Figure 2, when we test

Original: MLP-Mixer-B/16 as a target and the source network type matches the target, weights are identical.

Original: recordings with artificial neural network activations.

Original: By examining whether the candidate source model with the highest predictivity is identical to the target model, we can evaluate to what extent we can identify architectures with the current approach. 
Revised: By examining whether the candidate source model with the highest predictivity is identical to the target model, we can evaluate to what extent we can identify architectures with the current approach. [Figure]

Original: 4.2. L INEAR R EGRESSION
Revised: 4.2. L INEAR R EGRESSION

Original: We first compare various source models (AlexNet, VGG11, ResNet18,  ViT-B/32, MLP-Mixer-B/16) with a target network, the same architecture as one of the source models and is trained on the same dataset but initialized with a different seed.
Revised: We first compare various source models (AlexNet, VGG11, ResNet18, CORnet-S, ViT-B/32, MLPMixer-B/16) with a target network, the same architecture as one of the source models and is trained on the same dataset but initialized with a different seed.

Original: We test a dataset composed of 3200 images of synthetic objects studied in (Majaj et al., 2015) to be consistent with the evaluation pipeline of Brain-Score.
Revised: We test a dataset composed of 3200 images of synthetic objects studied in (Majaj et al., 2015) to be consistent with the evaluation pipeline of Brain-Score.

Original: The ground-truth source model will yield a higher score than other models if the model comparison pipeline is reliable.
Revised: The ground-truth source model will yield a higher score than other models if the model comparison pipeline is reliable.

Original: For most target layers,except those in VGG11, source layers with the highest  score are the matching layers in the same network (Figure 2).
Revised: For most target layers, source networks with the highest median score are the correct network (Figure 2).

Original: However, strikingly, for early and intermediate layers of target VGG11,the best-matched layers belong to a source model that is not VGG11.
Revised: However, strikingly, for several layers in VGG11,

Original: The correct source model outperforms other models only for three layers (two of which are the last two layers when close to the final classification task). The first layer of ResNet18 is also predicted best by ViT-B/32.
Revised: ResNet18, and CORnet-S the best-matched layers belong to a source model that is not the correct architecture.

Original: In other words, given the activations of VGG11, for instance, and based on linear regression scores, we would make an incorrect prediction that the system’s underlying architecture is closest to a ResNet18.
Revised: In other words, given the activations of VGG11, for instance, and based on linear regression scores, we would make an incorrect prediction that the system’s underlying architecture is closest to a ResNet18.

Original: In addition, because of our ideal setting, where an identical network is one of the source models, we expect to see a significant difference between matching and non-matching models.
Revised: In addition, because of our ideal setting, where an identical network is one of the source models, we expect to see a significant difference between matching and non-matching models.

Original: However, for some target layers in AlexNet and ResNet18,although the layer with the highest score may be the matching layer in the same architecture, linear regression scores for other source models do not show a significant decrease in predictivity.
Revised: However, for some target layers in VGG11 and ResNet18, linear regression scores for the non-identical architectures, when compared with those for the identical one, do not show a significant decrease in predictivity based on Welch’s t-test with p < 0 . 01 applied as a threshold (Figure 2).

Original: Taken together, this result suggests that the identification of the underlying architectures of unknown neural systems is far from perfect.
Revised: This result suggests that the identification of the underlying architectures of unknown neural systems is far from perfect.

Original: 4.2. CKA
Revised: 4.2. CKA

Original: Next, we examine another widely used metric, CKA, for comparing representations.
Revised: Next, we examine another widely used metric, CKA, for comparing representations.

Original: Again, we compare different source models to a target model, also an artificial neural network.
Revised: Again, we compare different source models to a target model, also an artificial neural network.

Original: For the target models we tested, the ground-truth source models achieved the highest score (Figure 3).
Revised: For the target models we tested, the ground-truth source models achieved the highest score (Figure 3).

Original: Still, some unmatched source networks lead to scores close to the matched networks, even for the target MLPMixer-B/16, where the source network of the same architecture type also has identical weights.
Revised: Still, some unmatched source networks lead to scores close to the matched networks, even for the target MLPMixer-B/16, where the source network of the same architecture type also has identical weights.

Revised: When applying CKA to compare representations, we subsample a set (3000) of target units to mimic the limited coverage of single-unit recordings.

Revised: Assuming we can increase the coverage for future

Original: [Figure]
Revised: [Figure]

Original: regression for different types of stimulus images and target networks. When applying CKA to compare representations, we subsample a set (3000) of target units to mimic the limited coverage of single-unit recordings.
Revised: CKA Identifiability Index [Figure]

Original: Assuming we can increase the coverage for future experiments with more advanced measurement techniques, we test whether the identifiability improves if we include all target units.
Revised: experiments with more advanced measurement techniques, we test whether the identifiability improves if we include all target units.

Original: Additionally, methods similar to CKA, such as RSA, are often applied to interpret neural recordings, including fMRI, MEG, and EEG (Cichy & Oliva, 2020; Cichy & Pantazis, 2017), which can have full coverage of the entire brain or a specific region of interest.
Revised: Additionally, methods similar to CKA, such as RSA, are often applied to interpret neural recordings, including fMRI, MEG, and EEG (Cichy & Oliva, 2020; Cichy & Pantazis, 2017), which can have full coverage of the entire brain or a specific region of interest.

Original: Therefore, we simulate such analyses by having all units in the targets.
Revised: Therefore, we simulate such analyses by having all units in the targets.

Original: Overall, the ground-truth source models outperform the other source models with a significant margin (Figure 7.
Revised: Overall, the ground-truth source models outperform the other source models with a significant margin (Figure 6.

Original: More experimental results are in the Appendix).
Revised: More experimental results are in the Appendix).

Original: This suggests system identification can be more reliable with more recording units and complete target coverage.
Revised: This suggests system identification can be more reliable with more recording units and complete target coverage.

Original: 4. E FFECTS OF THE STIMULUS DISTRIBUTION ON IDENTIFIABILITY
Revised: 4. E FFECTS OF THE STIMULUS DISTRIBUTION ON IDENTIFIABILITY

Original: A potentially significant variable overlooked in comparing computational models of the brain is the type of stimulus images.
Revised: A potentially significant variable overlooked in comparing computational models of the brain is the type of stimulus images.

Original: What types of stimulus images are suited for evaluating competing models?
Revised: What types of stimulus images are suited for evaluating competing models?

Original:  In Brain-Score, stimulus images for comparing models of the high-level visual areas, V4 and IT, are images of synthetic objects (Majaj et al., 2015).
Revised: [Figure] In Brain-Score, stimulus images for comparing models of the high-level visual areas, V4 and IT, are images of synthetic objects (Majaj et al., 2015).

Original: In contrast, those for the lower visual areas, V1 and
Revised: In contrast, those for the lower visual areas, V1 and

Original: V2, are images of texture and noise (Freeman et al., 2013).
Revised: V2, are images of texture and noise (Freeman et al., 2013).

Original: To examine the effect of using different stimulus images, we test images of synthetic objects (3200 images), texture and noise (135 images), and ImageNet (3000 images), which are more natural images than the first two datasets.
Revised: To examine the effect of using different stimulus images, we test images of synthetic objects (3200 images), texture and noise (135 images), and ImageNet (3000 images), which are more natural images than the first two datasets.

Original: In Figure 4, we analyze Identifiability Index for different stimulus images.
Revised: In Figure [Figure] we analyze Identifiability Index for different stimulus images.

Original: More realistic stimulus images (i.e., synthetic objects and ImageNet) show higher identifiability than texture and noise images for all target models.
Revised: More realistic stimulus images (i.e., synthetic objects and ImageNet) show higher identifiability than texture and noise images for all target models.

Original: For CKA, we observe identifiability increases with layer depth.
Revised: For CKA, we observe identifiability increases with layer depth.

Original: Notably, even for early layers in target models, which would correspond to V1 and V2 in the visual cortex, texture and noise images fail to give higher identifiability.
Revised: Notably, even for early layers in target models, which would correspond to V1 and V2 in the visual cortex, texture and noise images fail to give higher identifiability.

Original: Also, between images of synthetic objects and ImageNet, ImageNet shows higher identifiability.
Revised: Also, between images of synthetic objects and ImageNet, ImageNet shows higher identifiability.

Original: As target models are trained on ImageNet, our results suggest that using stimuli images closer to the images that targets see will help identify architectures better.
Revised: As target models are trained on ImageNet, our results suggest that using stimuli images closer to the images that targets see will help identify architectures better.

Original: It is important to note that the images of texture and noise we use in the experiment help characterize certain aspects of V1 and V2 in the previous work (Freeman et al., 2013).
Revised: It is important to note that the images of texture and noise we use in the experiment help characterize certain aspects of V1 and V2 in the previous work (Freeman et al., 2013).

Original: More specifically, the original work investigated the functional role of V2 in comparison with V1 by showing that naturalistic structure modulates V2.
Revised: More specifically, the original work investigated the functional role of V2 in comparison with V1 by showing that naturalistic structure modulates V2.

Original: Although the image set plays an influential variable in a carefully designed experiment for a more targeted hypothesis, it does not translate as a sufficient test set for any hypothesis, such as evaluating different neural networks.[-10 Target Layer Number 0. C K A Target: VGG13, 3000 Units/Layer        
-]
Revised: Although the image set plays an influential variable in a carefully designed experiment for a more targeted hypothesis, it does not translate as a sufficient test set for any hypothesis, such as evaluating different neural networks.[-10 Target Layer Number 0. C K A Target: VGG13, 3000 Units/Layer        
-]

Original: Target Layer Number

Original: Target: ResNet34, 3000 Units/Layer       21 Target Layer Number

Original: Target: ViT-L/16, 3000 Units/Layer p > 0.

Original: Model Family CNN ViT     10 Target Layer Number 0.

Original: R e g r e ss i o n

Original: S c

Original: o r e ( P e a r s o n ' s r )

Original: [-Target: VGG13, 3000 Units/Layer        
-]
Revised: [-Target: VGG13, 3000 Units/Layer        
-]

Original: Target Layer Number

Original: Target: ResNet34, 3000 Units/Layer       21 Target Layer Number

Original: Target: ViT-L/16, 3000 Units/Layer

Original: 4. C HALLENGES OF IDENTIFYING KEY ARCHITECTURAL MOTIFS
Revised: 4. C HALLENGES OF IDENTIFYING KEY ARCHITECTURAL MOTIFS

Original: Interesting hypotheses for a more biologically plausible design principle of brain-like models often involve key high-level architectural motifs.
Revised: Interesting hypotheses for a more biologically plausible design principle of brain-like models often involve key high-level architectural motifs.

Original: For instance, potential questions are whether recurrent connections are crucial in visual processing or, with the recent success of transformer models in deep learning, whether the brain similarly implements computations like attention layers in transformers.
Revised: For instance, potential questions are whether recurrent connections are crucial in visual processing or, with the recent success of transformer models in deep learning, whether the brain similarly implements computations like attention layers in transformers.

Original: The details beyond the key motif, such as the number of layers or exact type of activation functions, may vary and be underdetermined within the scope of such research questions.
Revised: The details beyond the key motif, such as the number of layers or exact type of activation functions, may vary and be underdetermined within the scope of such research questions.

Original: Likewise, it is unlikely that candidate models proposed by scientists align with the brain at every level, from lowlevel specifics to high-level computation.
Revised: Likewise, it is unlikely that candidate models proposed by scientists align with the brain at every level, from lowlevel specifics to high-level computation.

Original: Therefore, an ideal methodology for comparing models should help separate the key properties of interest while being invariant to other confounds.
Revised: Therefore, an ideal methodology for comparing models should help separate the key properties of interest while being invariant to other confounds.

Original: Considering it is a timely question, with the increased interest in transformers as models of the brain in different domains (Schrimpf et al., 2021; Berrios & Deza, 2022; Whittington et al., 2021), we focus on the problem of identifying convolution vs. attention.
Revised: Considering it is a timely question, with the increased interest in transformers as models of the brain in different domains (Schrimpf et al., 2021; Berrios & Deza, 2022; Whittington et al., 2021), we focus on the problem of identifying convolution vs. attention.

Original: We test 12 Convolutional Networks and 14 Vision Transformers of different architectures, and to maximize identifiability, we use ImageNet stimulus images.
Revised: We test 12 Convolutional Networks and 14 Vision Transformers of different architectures, and to maximize identifiability, we use ImageNet stimulus images.

Original: Note that an identical architecture with the target network is not included as a source network.
Revised: Note that an identical architecture with the target network is not included as a source network.

Original: Figure 5 shows that for most layers, mean CKA and regression scores are higher when there are correspondences between the target and source model classes than when there are not.
Revised: Figure 5 shows that for most layers, mean CKA and regression scores are higher when there are correspondences between the target and source model classes than when there are not.

Original: However, three layers in ViT-L/16 (layers 9, 15, 21) show a higher mean for the CNN model family than for the ViT model family.
Revised: However, three layers in ViT-L/16 (layers 9, 15, 21) show a higher mean for the CNN model family than for the ViT model family.

Original: Moreover, the architecture with the highest CKA score belongs to the CNN model family for three layers (layers 9-15).
Revised: Moreover, the architecture with the highest CKA score belongs to the CNN model family for three layers (layers 9-15).

Original: Overall, we observe high inter-class variance for many target layers for both methods.
Revised: Overall, we observe high inter-class variance for many target layers for both methods.

Original: For CKA, one layer in VGG13, 7 layers in ResNet34, and 7 layers in ViT-L/16, and for regression, three layers in VGG13, 7 layers in ResNet34, and one layer in ViT-L/16 do not show a statistically significant difference between the two model classes based on Welch’s t-test with p < 0 .
Revised: For CKA, one layer in VGG13, 7 layers in ResNet34, and 7 layers in ViT-L/16, and for regression, three layers in VGG13, 7 layers in ResNet34, and one layer in ViT-L/16 do not show a statistically significant difference between the two model classes based on Welch’s t-test with p < 0 .

Original: 01 used as a threshold.
Revised: 01 used as a threshold.

Original: The significant variance among source models suggests that model class identification can be incorrect depending on the precise variation we choose, especially if we rely on a limited set of models.
Revised: The significant variance among source models suggests that model class identification can be incorrect depending on the precise variation we choose, especially if we rely on a limited set of models.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Under idealized settings, we tested the identifiability of various artificial neural networks with differing architectures.
Revised: Under idealized settings, we tested the identifiability of various artificial neural networks with differing architectures.

Original: We present two contrasting interpretations of model identifiability based on our results, one optimistic (Glass half full) and one pessimistic (Glass half empty).
Revised: We present two contrasting interpretations of model identifiability based on our results, one optimistic (Glass half full) and one pessimistic (Glass half empty).

Original: Glass half full:
Revised: Glass half full:

Original: Despite the many factors that can lead to variable scores, linear regression and
Revised: Despite the many factors that can lead to variable scores, linear regression and

Original: CKA give reasonable identification capability under unrealistically ideal conditions.
Revised: CKA give reasonable identification capability under unrealistically ideal conditions.

Original: Across all the architectures tested, identifiability improves as a function of depth.
Revised: Across all the architectures tested, identifiability improves as a function of depth.

Original: Glass half empty: However, system identification is highly variable and dependent on the properties of the target architecture and the stimulus data used to probe the candidate models.
Revised: Glass half empty: However, system identification is highly variable and dependent on the properties of the target architecture and the stimulus data used to probe the candidate models.

Original: For architecturewide motifs, like convolution vs. attention, scores overlap significantly across almost all layers.
Revised: For architecturewide motifs, like convolution vs. attention, scores overlap significantly across almost all layers.

Original: This indicates that such distinct motifs do not play a significant role in the score.
Revised: This indicates that such distinct motifs do not play a significant role in the score.

Original: Our results suggest two future directions for improving system identification with current approaches: 1) Using stimuli images that are more natural, i.e., closer to the inputs to the target network (brain) in a natural setting.
Revised: Our results suggest two future directions for improving system identification with current approaches: 1) Using stimuli images that are more natural, i.e., closer to the inputs to the target network (brain) in a natural setting.

Original: With more neurons recorded in the brain, neural predictivity scores can be more reliable in finding the underlying architecture.
Revised: With more neurons recorded in the brain, neural predictivity scores can be more reliable in finding the underlying architecture.

Original: On the other hand, it is worthwhile to note that we may have reached close to the ceiling using neural predictivity scores for system identification.
Revised: On the other hand, it is worthwhile to note that we may have reached close to the ceiling using neural predictivity scores for system identification.

Original: As an example, when our source network is AlexNet, its regression scores against the brain (Figure 1) are on par with, or slightly higher than, the scores against another AlexNet (Figure 2).
Revised: As an example, when our source network is AlexNet, its regression scores against the brain (Figure 1) are on par with, or slightly higher than, the scores against another AlexNet (Figure 2).

Original: In other words, based on the current methods, AlexNet predicts the brain as well as, if not better than, predicting itself.
Revised: In other words, based on the current methods, AlexNet predicts the brain as well as, if not better than, predicting itself.

Original: This observation is not limited to AlexNet but applies to other target networks.
Revised: This observation is not limited to AlexNet but applies to other target networks.

Original: This fundamental limitation of present evaluation techniques, such as the linear encoding analysis used in isolation, emphasizes the need to develop new approaches beyond comparing functional similarities.
Revised: This fundamental limitation of present evaluation techniques, such as the linear encoding analysis used in isolation, emphasizes the need to develop new approaches beyond comparing functional similarities.

Original: As we argued earlier, ranking models in terms of their agreement with neural recordings is the first step in verifying or falsifying a neuroscience model.
Revised: As we argued earlier, ranking models in terms of their agreement with neural recordings is the first step in verifying or falsifying a neuroscience model.

Original: Since several different models are very close in ranking, the next step – architectural validation – is the key.
Revised: Since several different models are very close in ranking, the next step – architectural validation – is the key.

Original: Furthermore, it may have to be done independently of functional validation and with little guidance from it, using standard experimental tools in neuroscience.
Revised: Furthermore, it may have to be done independently of functional validation and with little guidance from it, using standard experimental tools in neuroscience.

Original: A parallel direction is, however, to try to develop specially designed, critical stimuli to distinguish between different architectures instead of measuring the overall fit to data.
Revised: A parallel direction is, however, to try to develop specially designed, critical stimuli to distinguish between different architectures instead of measuring the overall fit to data.

Original: As a simple example, it may be possible to discriminate between dense and local (e g., CNN) network architectures by measuring the presence or absence of interactions between parts of a visual stimulus that are spatially separated.
Revised: As a simple example, it may be possible to discriminate between dense and local (e.g., CNN) network architectures by measuring the presence or absence of interactions between parts of a visual stimulus that are spatially separated.

Original: M ODEL DETAILS FOR S ECTION 4.1: B RAIN -S CORE
Revised: M ODEL DETAILS FOR S ECTION 4.1: B RAIN -S CORE

Original: Below is the full list of models tested on the benchmarks of Brain-Score as reported in Section 4.1.
Revised: Below is the full list of models tested on the benchmarks of Brain-Score as reported in Section 4.1.

Original: In addition to testing vision models pre-trained on ImageNet available from PyTorch’s torchvision model package version 0.12, we test VOneNets that are pre-trained on ImageNet and made publicly available by the authors (Dapello et al., 2020).
Revised: In addition to testing vision models pre-trained on ImageNet available from PyTorch’s torchvision model package version 0.12, we test VOneNets that are pre-trained on ImageNet and made publicly available by the authors (Dapello et al., 2020).

Original: VOneNets are also a family of CNNs.
Revised: VOneNets are also a family of CNNs.

Original: Convolutional Networks: AlexNet, VGG11, VGG13, VGG19, ResNet18, ResNet34, ResNet50,
Revised: Convolutional Networks: AlexNet, VGG11, VGG13, VGG19, ResNet18, ResNet34, ResNet50,

Original: ResNet101, VOneAlexNet, VOneResNet50, VOnetCORnet-S
Revised: ResNet101, VOneAlexNet, VOneResNet50, VOnetCORnet-S

Original: Transformer Networks: ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/
Revised: Transformer Networks: ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/

Original: A.
Revised: A.

Original: M
Revised: M

Original: ODEL DETAILS FOR S ECTION 4.5: FINDING THE KEY ARCHITECTURAL MOTIF
Revised: ODEL DETAILS FOR S ECTION 4.5: FINDING THE KEY ARCHITECTURAL MOTIF

Original: For each target network reported in Section 4.5, namely VGG13, ResNet34, and ViT-L/16, below is the full list of source models tested to compare two model classes, CNN and transformer.
Revised: For each target network reported in Section 4.5, namely VGG13, ResNet34, and ViT-L/16, below is the full list of source models tested to compare two model classes, CNN and transformer.

Original: For
Revised: For

Original: Tokens-to-token ViTs (T2T) (Yuan et al., 2021), we use models released by the authors.
Revised: Tokens-to-token ViTs (T2T) (Yuan et al., 2021), we use models released by the authors.

Original: We use
Revised: We use

Original: Twins Vision Transformers and a Visformer from timm library (Wightman, 2019).
Revised: Twins Vision Transformers and a Visformer from timm library (Wightman, 2019).

Original: All other models are available from PyTorch’s torchvision model package version 0.12.
Revised: All other models are available from PyTorch’s torchvision model package version 0.12.

Original: All models are pre-trained on
Revised: All models are pre-trained on

Original: ImageNet.
Revised: ImageNet.

Original: Convolutional
Revised: Convolutional

Original: Networks :
Revised: Networks :

Original: AlexNet
Revised: AlexNet

Original: VGG11, VGG13, VGG16, VGG13 bn, ResNet18, ResNet34,
Revised: VGG11, VGG13, VGG16, VGG13 bn, ResNet18, ResNet34,

Original: ResNet50,
Revised: ResNet50,

Original: Wide
Revised: Wide

Original: -ResNet50 2,
Revised: -ResNet50 2,

Original: SqueezeNet
Revised: SqueezeNet

Original: Densenet
Revised: Densenet

Original: , MobileNet v
Revised: , MobileNet v

Original: Transfomer Networks: ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/32, T2T-ViT t-14, T2T-ViT t19, T2T-ViT
Revised: Transfomer Networks: ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/32, T2T-ViT t-14, T2T-ViT t19, T2T-ViT

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: A. R IDGE REGRESSION REGULARIZATION COEFFICIENT

Original: [Figure]
Revised: [Figure]

Original: i

Original: [Figure]
Revised: [Figure]

Original: c

Original: [Figure]
Revised: [Figure]

Original: r

Original: [Figure]
Revised: [Figure]

Original: i

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]