Original: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models
Revised: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Decades of experimental research based on simple, abstract stimuli has revealedthe coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complexvisual categories in the anterior ventral stream.
Revised: Decades of experimental research based on simple, abstract stimuli has revealedthe coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complexvisual categories in the anterior ventral stream.

Original: However, these studies are, byconstruction, constrained by their a priori hypotheses.
Revised: However, these studies are, byconstruction, constrained by their a priori hypotheses.

Original: Furthermore, beyond theearly stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.
Revised: Furthermore, beyond theearly stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.

Original: In this work, wepropose to employ response-optimized encoding models trained solely to predictthe functional MRI activation, in order to gain insights into the tuning propertiesand representational transformations in the series of areas along the ventral visualpathway.
Revised: In this work, wepropose to employ response-optimized encoding models trained solely to predictthe functional MRI activation, in order to gain insights into the tuning propertiesand representational transformations in the series of areas along the ventral visualpathway.

Original: We demonstrate the strong generalization abilities of these models onartificial stimuli and novel datasets.
Revised: We demonstrate the strong generalization abilities of these models onartificial stimuli and novel datasets.

Original: Intriguingly, we find that response-optimizedmodels trained towards the ventral-occipital and lateral-occipital areas, but notearly visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.
Revised: Intriguingly, we find that response-optimizedmodels trained towards the ventral-occipital and lateral-occipital areas, but notearly visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.

Original: We further probe the trainednetworks to reveal representational biases in different visual areas and generateexperimentally testable hypotheses.
Revised: We further probe the trainednetworks to reveal representational biases in different visual areas and generateexperimentally testable hypotheses.

Original:  Introduction
Revised:  Introduction

Original: Understanding the nature of representations and computations in the visual system has been alongstanding goal in neuroscience.
Revised: Understanding the nature of representations and computations in the visual system has been alongstanding goal in neuroscience.

Original: Substantial progress been made by presenting model organismswith simple, abstract stimuli like noise or sine-wave gratings and studying the evoked responses.
Revised: Substantial progress been made by presenting model organismswith simple, abstract stimuli like noise or sine-wave gratings and studying the evoked responses.

Original: These carefully crafted experiments, for instance, revealed the presence of edge detectors in theprimary visual cortex [1] and sparked the search for the ‘optimal input’ beyond early visual areasacross the visual cortical hierarchy.
Revised: These carefully crafted experiments, for instance, revealed the presence of edge detectors in theprimary visual cortex [1] and sparked the search for the ‘optimal input’ beyond early visual areasacross the visual cortical hierarchy.

Original: Significant efforts have also been made to understand the featurecomplexity of tuning in V4 and IT neurons [2, 3, 4, 5].
Revised: Significant efforts have also been made to understand the featurecomplexity of tuning in V4 and IT neurons [2, 3, 4, 5].

Original: Over the last two decades, through numerousexperiments, recording and analysis techniques, a conceptual understanding has emerged that earlyvisual areas extract low-level features like edges or curves, mid-level regions extract complex localshapes, and high-level regions encode semantic categories, like faces or scenes [6, 7, 8, 9, 10].
Revised: Over the last two decades, through numerousexperiments, recording and analysis techniques, a conceptual understanding has emerged that earlyvisual areas extract low-level features like edges or curves, mid-level regions extract complex localshapes, and high-level regions encode semantic categories, like faces or scenes [6, 7, 8, 9, 10].

Original: Whilemuch of the details remain unknown, there is accumulating evidence that visual categorization occursin the ventral temporal cortex in humans [11].
Revised: Whilemuch of the details remain unknown, there is accumulating evidence that visual categorization occursin the ventral temporal cortex in humans [11].

Original: This understanding was enabled by synthesizingfindings across different hypothesis-driven studies, most of them employing artificial stimuli atypicalof real-world vision, thus lacking ethological validity.
Revised: This understanding was enabled by synthesizingfindings across different hypothesis-driven studies, most of them employing artificial stimuli atypicalof real-world vision, thus lacking ethological validity.

Original: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks,have emerged as the most accurate models of the primate ventral visual stream [12, 13, 14, 15].
Revised: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks,have emerged as the most accurate models of the primate ventral visual stream [12, 13, 14, 15].

Original: Furthermore, these CV task-optimized models offer a hierarchical correspondence to visual areasalong the ventral stream: early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [12, 16, 17, 18, 19, 20, 21, 22].
Revised: Furthermore, these CV task-optimized models offer a hierarchical correspondence to visual areasalong the ventral stream: early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [12, 16, 17, 18, 19, 20, 21, 22].

Original: They provide intelligibility in the formof the goal-driven perspective, highlighting the functional role of representations [23].
Revised: They provide intelligibility in the formof the goal-driven perspective, highlighting the functional role of representations [23].

Original: However,we are far from a complete understanding of the top-down computational objectives (the batteryof ethologically-relevant tasks) that may have shaped neural representations through evolution orexperience-based learning; in the absence of right hypotheses, the task-optimized approach can fallshort.
Revised: However,we are far from a complete understanding of the top-down computational objectives (the batteryof ethologically-relevant tasks) that may have shaped neural representations through evolution orexperience-based learning; in the absence of right hypotheses, the task-optimized approach can fallshort.

Original: Studies have already revealed significant inconsistencies between these models and humanbehavior and gaps remain in their respective robustness to distortions, image-level consistencyetc [24].
Revised: Studies have already revealed significant inconsistencies between these models and humanbehavior and gaps remain in their respective robustness to distortions, image-level consistencyetc [24].

Original: Recently, the idea of using neural data in the model development process has gainedsome traction, and demonstrated success in increasing both model-human behavior [25, 26, 27],and model-brain alignment, while revealing novel insights about information processing in neuralsystems [28, 29, 30, 31].
Revised: Recently, the idea of using neural data in the model development process has gainedsome traction, and demonstrated success in increasing both model-human behavior [25, 26, 27],and model-brain alignment, while revealing novel insights about information processing in neuralsystems [28, 29, 30, 31].

Original: In this study, we leverage recent advances in large-scale fMRI data collection [32] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.
Revised: In this study, we leverage recent advances in large-scale fMRI data collection [32] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.

Original: Importantly,the stimulus set employed in this study contains crowded images of multiple objects in their natural contexts, thus being more typical of everyday scenes and allowing us to characterize neuralrepresentations and computations in rich, naturalistic conditions.
Revised: Importantly,the stimulus set employed in this study contains crowded images of multiple objects in their natural contexts, thus being more typical of everyday scenes and allowing us to characterize neuralrepresentations and computations in rich, naturalistic conditions.

Original: We demonstrate the remarkableability of these “response-optimized” neural encoding models in reproducing the stimulus-responserelationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassingthe unprecedented prediction performance of task-optimized models.
Revised: We demonstrate the remarkableability of these “response-optimized” neural encoding models in reproducing the stimulus-responserelationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassingthe unprecedented prediction performance of task-optimized models.

Original: We show that these modelsgeneralize well to contexts that are vastly different from their training conditions (artificial stimuli,novel subjects, etc.), providing a strong quantitative validation for the response-optimization approachof building encoding models for neurobiological visual systems beyond the retina [33].
Revised: We show that these modelsgeneralize well to contexts that are vastly different from their training conditions (artificial stimuli,novel subjects, etc.), providing a strong quantitative validation for the response-optimization approachof building encoding models for neurobiological visual systems beyond the retina [33].

Original: Our approachfurther goes beyond prediction accuracy and aims to gain insights about neuronal tuning properties.
Revised: Our approachfurther goes beyond prediction accuracy and aims to gain insights about neuronal tuning properties.

Original: Specifically, we implement neural encoding models that disentangle the processing of the “what” andthe “where.”
Revised: Specifically, we implement neural encoding models that disentangle the processing of the “what” andthe “where.”

Original: This allows us to reveal the selectivity of individual brain voxels along both locationand appearance dimensions.
Revised: This allows us to reveal the selectivity of individual brain voxels along both locationand appearance dimensions.

Original: We first demonstrate that the estimated spatial topographic organizationfrom our neural encoding models exhibits remarkable agreement with the fine-scaled retinotopicorganization of the cortex, which is typically measured with dedicated, painstaking experimentalparadigms.
Revised: We first demonstrate that the estimated spatial topographic organizationfrom our neural encoding models exhibits remarkable agreement with the fine-scaled retinotopicorganization of the cortex, which is typically measured with dedicated, painstaking experimentalparadigms.

Original: We analyze the features that emerge in the neural encoding models to understand the tuning properties of cortical voxels.
Revised: We analyze the features that emerge in the neural encoding models to understand the tuning properties of cortical voxels.

Original: This analysis suggests shape-biased tuning across the ventral visualstream, increasing in complexity from Gabor-like features in V1, curves/angles in V2 to concentricand complex shape processing in V4 and higher-order lateral-occipital and ventral-occipital ROIsrespectively.
Revised: This analysis suggests shape-biased tuning across the ventral visualstream, increasing in complexity from Gabor-like features in V1, curves/angles in V2 to concentricand complex shape processing in V4 and higher-order lateral-occipital and ventral-occipital ROIsrespectively.

Original: We also go beyond single-voxel analysis to characterize how representations evolvealong the ventral visual pathway.
Revised: We also go beyond single-voxel analysis to characterize how representations evolvealong the ventral visual pathway.

Original: Probing the information encoded in model-predicted patterns ofactivity reveals the monotonically increasing separability of category information along the ventralstream hierarchy, strongly supporting the computational goal of the ventral visual stream as a visualcategorization system.
Revised: Probing the information encoded in model-predicted patterns ofactivity reveals the monotonically increasing separability of category information along the ventralstream hierarchy, strongly supporting the computational goal of the ventral visual stream as a visualcategorization system.

Original: We further demonstrate the correspondence between model representations (ofhigh-level visual areas) and mental representations (as assessed with human similarity judgements),suggesting that these response-optimized models inherently capture human psychological judgements,without any explicit supervision to do so.
Revised: We further demonstrate the correspondence between model representations (ofhigh-level visual areas) and mental representations (as assessed with human similarity judgements),suggesting that these response-optimized models inherently capture human psychological judgements,without any explicit supervision to do so.

Original: Furthermore, a principle components analysis reveals thatmodels of high-level visual ROIs capture complex stimulus dimensions like faces, bodies and shapes.
Revised: Furthermore, a principle components analysis reveals thatmodels of high-level visual ROIs capture complex stimulus dimensions like faces, bodies and shapes.

Original: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, naturalistic datasetscan be used to confirm neural phenomena characterized painstakingly using controlled experimentalparadigms that might be hard to replicate in individual subjects.
Revised: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, naturalistic datasetscan be used to confirm neural phenomena characterized painstakingly using controlled experimentalparadigms that might be hard to replicate in individual subjects.

Original: More importantly, however, we showhow in silico experiments on neural encoding models can generate novel hypotheses and provide apowerful framework for building broad, generalizable theories about neural information processing.
Revised: More importantly, however, we showhow in silico experiments on neural encoding models can generate novel hypotheses and provide apowerful framework for building broad, generalizable theories about neural information processing.

Original:  Materials and Methods
Revised:  Materials and Methods

Original: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is providedelsewhere [32] (see also the Appendix).
Revised: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is providedelsewhere [32] (see also the Appendix).

Original: The dataset contains measurements of fMRI responses from8 participants who each viewed 9,000–10,000 natural scenes.
Revised: The dataset contains measurements of fMRI responses from8 participants who each viewed 9,000–10,000 natural scenes.

Original: A special set of 1,000 images wereshared across subjects; the remaining were mutually exclusive.
Revised: A special set of 1,000 images wereshared across subjects; the remaining were mutually exclusive.

Original: We focused on modeling responseswithin 5 visual cortical ROIs in the study.
Revised: We focused on modeling responseswithin 5 visual cortical ROIs in the study.

Original: Three ROIs belonging to the retinoptic early visual cortex,namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session [32].
Revised: Three ROIs belonging to the retinoptic early visual cortex,namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session [32].

Original: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipitalareas (LO1-2) were delineated using a visual probabilistic atlas [34](Figure 1B).
Revised: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipitalareas (LO1-2) were delineated using a visual probabilistic atlas [34](Figure 1B).

Original: Response-optimized neural encoding model
Revised: Response-optimized neural encoding model

Original: We trained separate voxel-level neural encodingmodels for each of the above regions with the same backbone architecture.
Revised: We trained separate voxel-level neural encodingmodels for each of the above regions with the same backbone architecture.

Original: Keeping the architectureconstant across visual areas ensured that differences in emergent representations are driven mainlyby the nature of response patterns.
Revised: Keeping the architectureconstant across visual areas ensured that differences in emergent representations are driven mainlyby the nature of response patterns.

Original: The predictive model comprises a convolutional neural network(CNN) feature extractor shared across all subjects, but unique to specific visual areas.
Revised: The predictive model comprises a convolutional neural network(CNN) feature extractor shared across all subjects, but unique to specific visual areas.

Original: We employ alinear readout model on top of the feature space to predict the responses of individual voxels in aspecific region.
Revised: We employ alinear readout model on top of the feature space to predict the responses of individual voxels in aspecific region.

Original: The linear readout is factorized into spatial and feature dimensions following [35](Figure 1A).
Revised: The linear readout is factorized into spatial and feature dimensions following [35](Figure 1A).

Original: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).
Revised: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).

Original: The shared CNN feature extractor consists of four convolutional blocks, with each block comprisingthe following feedforward computations: two 3 × 3 convolutional layers, each followed by an innerbatch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.
Revised: The shared CNN feature extractor consists of four convolutional blocks, with each block comprisingthe following feedforward computations: two 3 × 3 convolutional layers, each followed by an innerbatch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.

Original: We employ
Revised: We employ

Original: E(2)-steerable convolutions in all our models to extract features invariant to orientation [36].
Revised: E(2)-steerable convolutions in all our models to extract features invariant to orientation [36].

Original: Thismodeling choice is inspired by neural computations in early visual areas where groups of neurons areknown to perform similar computations, e.g., edge or curve detection, at different orientations.
Revised: Thismodeling choice is inspired by neural computations in early visual areas where groups of neurons areknown to perform similar computations, e.g., edge or curve detection, at different orientations.

Original: Eachconvolutional layer contains 48 feature sets extracted at 8 orientations.
Revised: Eachconvolutional layer contains 48 feature sets extracted at 8 orientations.

Original: Weights of the readout are acomputed as outer products between a spatial filter and a feature vector.
Revised: Weights of the readout are acomputed as outer products between a spatial filter and a feature vector.

Original: The spatial filter further hasa positivity constraint (enforced using rectification) and is length-normalized for each voxel.
Revised: The spatial filter further hasa positivity constraint (enforced using rectification) and is length-normalized for each voxel.

Original: Training and testing response-optimized encoding models We used 4 NSD subjects for trainingand reserved the other 4 subjects for studying generalization.
Revised: Training and testing response-optimized encoding models We used 4 NSD subjects for trainingand reserved the other 4 subjects for studying generalization.

Original: The first group collectively saw 37,000natural scene images, including the 1,000 shared images.
Revised: The first group collectively saw 37,000natural scene images, including the 1,000 shared images.

Original: We used the 1,000 shared images for testingour models and split the remaining stimulus set into 35,000 training and 2,000 validation images.
Revised: We used the 1,000 shared images for testingour models and split the remaining stimulus set into 35,000 training and 2,000 validation images.

Original: All parameters of the neural encoding model were optimized jointly to minimize the masked meansquared error between the predicted and measured fMRI response.
Revised: All parameters of the neural encoding model were optimized jointly to minimize the masked meansquared error between the predicted and measured fMRI response.

Original: This loss allows us to propagateerrors through the shared CNN even if the subjects are not exposed to common stimuli since we canexclude the subjects/voxels for which the target response is not present from the loss calculation.
Revised: This loss allows us to propagateerrors through the shared CNN even if the subjects are not exposed to common stimuli since we canexclude the subjects/voxels for which the target response is not present from the loss calculation.

Original: Wequantify ‘predictive accuracy’ on the test images as the Pearson’s correlation coefficient between thepredicted and measured fMRI response at each voxel.
Revised: Wequantify ‘predictive accuracy’ on the test images as the Pearson’s correlation coefficient between thepredicted and measured fMRI response at each voxel.

Original: We also compare this against the noise-ceiling,expressed as the signal-to-noise ratio when considering the response variability of each voxel across3 repetitions per image (see Appendix A.5).
Revised: We also compare this against the noise-ceiling,expressed as the signal-to-noise ratio when considering the response variability of each voxel across3 repetitions per image (see Appendix A.5).

Original: Table A.1 provides a summary of all analyses methodsand baselines used for evaluating and interpreting the proposed response-optimized models.
Revised: Table A.1 provides a summary of all analyses methodsand baselines used for evaluating and interpreting the proposed response-optimized models.

Original: Baseline models (a) Category ideal observer model: We fit a non-image-computable modelusing human-generated annotations for NSD images (obtained from the MS-COCO database).
Revised: Baseline models (a) Category ideal observer model: We fit a non-image-computable modelusing human-generated annotations for NSD images (obtained from the MS-COCO database).

Original: Theinput to the categorical model is a k-hot encoding vector corresponding to the 80 object categoriesannotated in the database, where each element indicates the presence/absence of the correspondingcategory in the image.
Revised: Theinput to the categorical model is a k-hot encoding vector corresponding to the 80 object categoriesannotated in the database, where each element indicates the presence/absence of the correspondingcategory in the image.

Original: We fitted l 2 regularized linear regression models on this space to predictvoxel responses for held-out stimuli (Appendix A.8).
Revised: We fitted l 2 regularized linear regression models on this space to predictvoxel responses for held-out stimuli (Appendix A.8).

Original: (b) Task-optimized Models: We investigatedseveral DNN models, including AlexNet [37], ResNet-50 [38], DenseNet [39] and CORnet-s [40],optimized for object recognition on the large-scale ImageNet dataset [41] (Appendix A.3, Table
Revised: (b) Task-optimized Models: We investigatedseveral DNN models, including AlexNet [37], ResNet-50 [38], DenseNet [39] and CORnet-s [40],optimized for object recognition on the large-scale ImageNet dataset [41] (Appendix A.3, Table

Original: A.2).
Revised: A.2).

Original: Since these models can demonstrate lower prediction performance due to the domain shiftbetween ImageNet and NSD stimuli, we also employed a ResNet-50 architecture pre-trained forobject detection on the MS-COCO dataset [42] as an additional baseline (since NSD stimuli weredrawn from the MS-COCO dataset).
Revised: Since these models can demonstrate lower prediction performance due to the domain shiftbetween ImageNet and NSD stimuli, we also employed a ResNet-50 architecture pre-trained forobject detection on the MS-COCO dataset [42] as an additional baseline (since NSD stimuli weredrawn from the MS-COCO dataset).

Original: For CORnet-S, we pre-select the layer homologous to eachvisual region and employ a spatial × feature factorized readout (same as the response-optimizedmodel) on top of the fixed feature backbone, allowing a more direct comparison to the proposed model.
Revised: For CORnet-S, we pre-select the layer homologous to eachvisual region and employ a spatial × feature factorized readout (same as the response-optimizedmodel) on top of the fixed feature backbone, allowing a more direct comparison to the proposed model.

Original: For the remaining models, we followed the more standard procedure for fitting task-optimizedmodels to voxel responses based on a full linear readout.
Revised: For the remaining models, we followed the more standard procedure for fitting task-optimizedmodels to voxel responses based on a full linear readout.

Original: We extracted features from pre-trainedlayers of each network and fitted l 2 regularized linear regression models separately for each layer.
Revised: We extracted features from pre-trainedlayers of each network and fitted l 2 regularized linear regression models separately for each layer.

Original: The regularization parameter was optimized independently for each subject, each layer, each modeland for voxels in each visual area by testing among 8 log-spaced values in [1e-4, 1e4].
Revised: The regularization parameter was optimized independently for each subject, each layer, each modeland for voxels in each visual area by testing among 8 log-spaced values in [1e-4, 1e4].

Original: For each visualarea and model, we identified the layer that best predicts responses on the validation set.
Revised: For each visualarea and model, we identified the layer that best predicts responses on the validation set.

Original: These modellayers were used to quantify test performance.
Revised: These modellayers were used to quantify test performance.

Original: (c) Response-optimized rotation symmetries):
Revised: (c) Response-optimized rotation symmetries):

Original: Here, we train response-optimized models with the same architecture as the proposed models butwithout sharing weights across filter orientations.
Revised: Here, we train response-optimized models with the same architecture as the proposed models butwithout sharing weights across filter orientations.

Original: Characterizing neural representations by behavioral comparisons: (a) Assessing separabilityof category and semantic information along the visual hierarchy To probe the learned featuraltuning in the response-optimized model for each visual area, we used representational similarityanalysis (RSA) [43].
Revised: Characterizing neural representations by behavioral comparisons: (a) Assessing separabilityof category and semantic information along the visual hierarchy To probe the learned featuraltuning in the response-optimized model for each visual area, we used representational similarityanalysis (RSA) [43].

Original: Here, we correlated the patterns of predicted activity between multipleexemplars from different categories of stimuli to obtain a representational similarity matrix (RSM)for each visual area.
Revised: Here, we correlated the patterns of predicted activity between multipleexemplars from different categories of stimuli to obtain a representational similarity matrix (RSM)for each visual area.

Original: Further, we quantified the separability of categorical information within thesesimilarity matrices by computing a correlation coefficient (Kendall’s τ ) between model RSMs againsta ground truth adjacency matrix defined by object category labels.
Revised: Further, we quantified the separability of categorical information within thesesimilarity matrices by computing a correlation coefficient (Kendall’s τ ) between model RSMs againsta ground truth adjacency matrix defined by object category labels.

Original: Specifically, the elements of thismatrix are 1 if the corresponding two images belong to different categories and 0 if the imagesbelong to the same category.
Revised: Specifically, the elements of thismatrix are 1 if the corresponding two images belong to different categories and 0 if the imagesbelong to the same category.

Original: This analysis was first performed using a subset of 104 images fromthe THINGS database [44] that belong to a pre-defined set of categories for ease of visualization.
Revised: This analysis was first performed using a subset of 104 images fromthe THINGS database [44] that belong to a pre-defined set of categories for ease of visualization.

Original: These categories were chosen in accordance with previous fMRI studies employing RSA [45, 46]and include the following: { face, hand, elephant, cat, plant, fruit, car, tool } .
Revised: These categories were chosen in accordance with previous fMRI studies employing RSA [45, 46]and include the following: { face, hand, elephant, cat, plant, fruit, car, tool } .

Original: Subsequently, we alsovalidated the obtained results using another large-scale vision dataset, namely ImageNet-16 [47].
Revised: Subsequently, we alsovalidated the obtained results using another large-scale vision dataset, namely ImageNet-16 [47].

Original: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images andcomputed the similarity between model RSMs and the categorical adjacency matrix following thesame approach described for the THINGS dataset.
Revised: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images andcomputed the similarity between model RSMs and the categorical adjacency matrix following thesame approach described for the THINGS dataset.

Original: To assess visual information flow in relation tohigh-level semantic understanding, we computed pairwise similarity between predicted responses forheld-out NSD images to construct model-predicted RSMs for each visual area.
Revised: To assess visual information flow in relation tohigh-level semantic understanding, we computed pairwise similarity between predicted responses forheld-out NSD images to construct model-predicted RSMs for each visual area.

Original: We constructed asemantic RSM by correlating the 80-D binary category vectors (which indicate whether each categorywas present in the image or not) for every pair of these NSD images and quantified agreement withmodel-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).
Revised: We constructed asemantic RSM by correlating the 80-D binary category vectors (which indicate whether each categorywas present in the image or not) for every pair of these NSD images and quantified agreement withmodel-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).

Original: Wecomputed this agreement using measured RSMs for all visual areas as well.
Revised: Wecomputed this agreement using measured RSMs for all visual areas as well.

Original: In RSA described hereand anywhere else, error bars (95 % CI) are computed by performing bootstrap sampling over voxels.
Revised: In RSA described hereand anywhere else, error bars (95 % CI) are computed by performing bootstrap sampling over voxels.

Original: (b) Assessing alignment with human perception We compared model representations againstbehavioral data from a Multiple Arrangement task where participants interpreted 100 NSD imagesand arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ wasreleased with NSD [32]).
Revised: (b) Assessing alignment with human perception We compared model representations againstbehavioral data from a Multiple Arrangement task where participants interpreted 100 NSD imagesand arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ wasreleased with NSD [32]).

Original: Critically, these images were not seen by any of the response-optimized models.
Revised: Critically, these images were not seen by any of the response-optimized models.

Original: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROIusing the correlation distance metric for quantifying pairwise dissimilarities between model-predictedvoxel responses.
Revised: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROIusing the correlation distance metric for quantifying pairwise dissimilarities between model-predictedvoxel responses.

Original: We compared model RDMs against human pairwise dissimilarity judgements usingthe Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.
Revised: We compared model RDMs against human pairwise dissimilarity judgements usingthe Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.

Original:  Results
Revised:  Results

Original: Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B,
Revised: Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B,

Original: Appendix Figure A.2, A.6).
Revised: Appendix Figure A.2, A.6).

Original: Next, we investigated the possible quantitative advantages of learninga rotationally-symmetric feature space against the representations from a standard convolutionalarchitecture with no weight sharing across orientations.
Revised: Next, we investigated the possible quantitative advantages of learninga rotationally-symmetric feature space against the representations from a standard convolutionalarchitecture with no weight sharing across orientations.

Original: Importantly, we observe that the proposedrotationally-symmetric convolutional architecture performs on par with a standard CNN (Figure 2A),even outperforming the latter in certain visual ROIs, despite containing fewer parameters.
Revised: Importantly, we observe that the proposedrotationally-symmetric convolutional architecture performs on par with a standard CNN (Figure 2A),even outperforming the latter in certain visual ROIs, despite containing fewer parameters.

Original: Thissuggests that sharing weights across filter orientations provides a strong inductive bias, allowing usto fit expressive models efficiently.
Revised: Thissuggests that sharing weights across filter orientations provides a strong inductive bias, allowing usto fit expressive models efficiently.

Original: We further compared these models against ideal observer andtask-optimized models, and found that response-optimized models consistently outperform thesebaselines in early visual areas.
Revised: We further compared these models against ideal observer andtask-optimized models, and found that response-optimized models consistently outperform thesebaselines in early visual areas.

Original: In high-level visual areas, the response-optimized model significantly outperforms the ideal observer model, while performing on par with or even surpassing sometask-optimized models.
Revised: In high-level visual areas, the response-optimized model significantly outperforms the ideal observer model, while performing on par with or even surpassing sometask-optimized models.

Original: Next, we assessed how each of these predictivemodels generalizes to the remaining set of 4test subjects.
Revised: Next, we assessed how each of these predictivemodels generalizes to the remaining set of 4 testsubjects.

Original: This analysis intended to comparethe statistical efficiency afforded by differentrepresentations [48].
Revised: This analysis intended to compare thestatistical efficiency afforded by different representations [48].

Original: As such, we examinedthe trend of how much stimulus-response datathe models need to achieve a certain level ofprediction performance [48].
Revised: As such, we examined the trendof how much stimulus-response data the models need to achieve a certain level of predictionperformance [48].

Original: Here, we freeze the shared feature extractor network and vary the amount of stimulus-responsepairs from the 4 new subjects to train the linear readouts (Appendix A.7).
Revised: Here, we freeze the sharedfeature extractor network and vary the amountof stimulus-response pairs from the 4 new subjects to train the linear readouts (Appendix A.7).

Original: We do the samefor the task-optimized baselines, where the linear prediction model is fit on new subjects.
Revised: We do the same for the task-optimized baselines,where the linear prediction model is fit on newsubjects.

Original: Asshown in Figure 2, the response-optimized models can achieve a significantly high accuracyeven when only just 5 % of the training examplesfrom novel subjects ( ∼ 500 samples) are usedto train the readout.
Revised: As shown in Figure 2, the responseoptimized models can achieve a significantlyhigh accuracy even when only just 5 % of thetraining examples from novel subjects ( ∼ 500samples) are used to train the readout.

Original: This remarkable generalization of response-optimized networks, exceedingthat of task-optimized networks, to novel subjects suggests our models contain appropriate“inductive biases" that improve sample efficiency.
Revised: This remarkable generalization of response-optimizednetworks, exceeding that of task-optimized networks, to novel subjects suggests our modelscontain appropriate “inductive biases" that improve sample efficiency.

Original: Computational models generalize to OOD datasets We next investigated out-of-distribution(OOD) test settings to understand the generalization behavior of response-optimized models.
Revised: Computational models generalize to OOD datasets We next investigated out-of-distribution(OOD) test settings to understand the generalization behavior of response-optimized models.

Original: Thefirst OOD setting we consider is the NSD-synthetic experiment, comprising fMRI recordings tosynthetic stimuli (samples shown in Figure 3) from the same NSD subjects.
Revised: Thefirst OOD setting we consider is the NSD-synthetic experiment, comprising fMRI recordings tosynthetic stimuli (samples shown in Figure 3) from the same NSD subjects.

Original: As shown in Figure 3,response-optimized models significantly outperform task-optimized models (except in region VO)in this OOD domain (‘artificial’ stimuli), suggesting that the mapping from task-optimized DNNfeatures to voxel responses, was domain/dataset-specific.
Revised: As shown in Figure 3,response-optimized models significantly outperform task-optimized models (except in region VO)in this OOD domain (‘artificial’ stimuli), suggesting that the mapping from task-optimized DNNfeatures to voxel responses, was domain/dataset-specific.

Original: Response-optimized models, on the otherhand, allow for flexible OOD generalization.
Revised: Response-optimized models, on the otherhand, allow for flexible OOD generalization.

Original: Generalization to other novel fMRI datasets, including[49, 50, 51] is shown and discussed in the Appendix B.2.
Revised: Generalization to other novel fMRI datasets, including[49, 50, 51] is shown and discussed in the Appendix B.2.

Original: Learned spatial masks reproduce the fine-grained retinotopic organization of early visual cortex
Revised: Learned spatial masks reproduce the fine-grained retinotopic organization of early visual cortex

Original: The factorized readout in the proposed model allows us to separate spatial and feature selectivity.
Revised: The factorized readout in the proposed model allows us to separate spatial and feature selectivity.

Original: We characterized the spatial tuning of early visual areas by estimating the population receptive field(pRF) from the learned spatial masks of the proposed response-optimized model; we compare theestimated parameters against measurements from the independent NSD pRF localizer experimentthat uses artificial stimuli.
Revised: We characterized the spatial tuning of early visual areas by estimating the population receptive field (pRF) from the learned spatial masks of the proposed response-optimized model; we compare theestimated parameters against measurements from the independent NSD pRF localizer experimentthat uses artificial stimuli.

Original: The pRF is defined as the region of the visual field within which a stimulusresults in increased aggregated activity across a population of neurons, as reflected in fMRI.
Revised: The pRF is defined as the region of the visual field within which a stimulusresults in increased aggregated activity across a population of neurons, as reflected in fMRI.

Original: Basedon the analysis of these data, each brain voxel’s retinotopic organization can be described with threeparameters: the pRF location in polar coordinates (polar angle and eccentricity), and size.
Revised: Basedon the analysis of these data, each brain voxel’s retinotopic organization can be described with threeparameters: the pRF location in polar coordinates (polar angle and eccentricity), and size.

Original: Usingthe voxel-wise spatial mask of the readout in our encoding model, we estimated the pRF location asthe average of the coordinates weighted by the learned spatial mask value at each grid location.
Revised: Usingthe voxel-wise spatial mask of the readout in our encoding model, we estimated the pRF location asthe average of the coordinates weighted by the learned spatial mask value at each grid location.

Original: Tocompare the estimated eccentricity against the corresponding measurements from pRF localizers,henceforth called ‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient betweenthe measured and predicted eccentricity arrays in each ROI.
Revised: Tocompare the estimated eccentricity against the corresponding measurements from pRF localizers,henceforth called ‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient betweenthe measured and predicted eccentricity arrays in each ROI.

Original: To compare the estimated polar anglevalues against corresponding measurements from pRF localizer scans, henceforth called ‘polar angleagreement’ , we adopted a measure from circular statistics called circular correlation coefficient [52],due to the circular nature of angular data (Appendix A.4).
Revised: To compare the estimated polar anglevalues against corresponding measurements from pRF localizer scans, henceforth called ‘polar angleagreement’ , we adopted a measure from circular statistics called circular correlation coefficient [52],due to the circular nature of angular data (Appendix A.4).

Original: It is important to note that the encodingmodels do not account for the spatial locations of target voxels.
Revised: It is important to note that the encodingmodels do not account for the spatial locations of target voxels.

Original: Despite this, the estimated spatialtopographic organization from the encoding models exhibits remarkable levels of agreement withthe fine-scaled retinotopic organization of the cortex without any explicit supervision to do so(Figure
Revised: Despite this, the estimated spatialtopographic organization from the encoding models exhibits remarkable levels of agreement withthe fine-scaled retinotopic organization of the cortex without any explicit supervision to do so(Figure

Original: 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angleagreement ∈ [0.70-0.82].
Revised: 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angleagreement ∈ [0.70-0.82].

Original: We further compared this agreement on the estimated pRF parametersfor the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linearreadouts from the fixed convolutional core of the response-optimized model (pre-trained on theinitial set of 4 subjects).
Revised: We further compared this agreement on the estimated pRF parametersfor the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linearreadouts from the fixed convolutional core of the response-optimized model (pre-trained on theinitial set of 4 subjects).

Original: We find that while the agreement increases with the sample size used tofit readouts, even with much fewer samples (e.g. 100), we can still characterize the retinotopicorganization (Figure 4C).
Revised: We find that while the agreement increases with the sample size used tofit readouts, even with much fewer samples (e.g. 100), we can still characterize the retinotopicorganization (Figure 4C).

Original: This is better than the retinotopic agreement observed for the CORnetS model (which also employed a factorized readout, enabling characterization of spatial tuning(Appendix Figure A.8)).
Revised: This is better than the retinotopic agreement observed for the CORnetS model (which also employed a factorized readout, enabling characterization of spatial tuning(Appendix Figure A.8)).

Original: Previously, retinotopic organization of the human visual cortex has beendominantly studied with spatially modulated stimuli.
Revised: Previously, retinotopic organization of the human visual cortex has beendominantly studied with spatially modulated stimuli.

Original: While encoding models have been shown tobe broadly useful for characterizing retinotopy [53, 31], a precise quantification of the agreementbetween model-estimated and localizer-estimated retinotopic parameters, as well as the effect ofsample sizes on this agreement, has been lacking.
Revised: While encoding models have been shown tobe broadly useful for characterizing retinotopy [53, 31], a precise quantification of the agreementbetween model-estimated and localizer-estimated retinotopic parameters, as well as the effect ofsample sizes on this agreement, has been lacking.

Original: Here, we present an alternate approach basedon naturalistic stimulation and response-optimized encoding models to delineate subject-specificretinotopic maps and precisely quantify the agreement.
Revised: Here, we present an alternate approach basedon naturalistic stimulation and response-optimized encoding models to delineate subject-specificretinotopic maps and precisely quantify the agreement.

Original: Remarkably, the retintopic organizationestimated from the proposed encoding models is even more spatially smooth than the localizer-basedorganization, despite not being informed by anatomy.
Revised: Remarkably, the retintopic organizationestimated from the proposed encoding models is even more spatially smooth than the localizer-basedorganization, despite not being informed by anatomy.

Original: Response-optimized models characterize transformations of image representations along thevisual cortical hierarchy
Revised: visual cortical hierarchy

Original: (a) Monotonically increasing alignment with human perception along the ventral visual stream
Revised: (a) Monotonically increasing alignment with human perception along the ventral visual stream

Original: We evaluated models on behavioral data from a Multiple Arrangement task where participantsarranged NSD images according to the similarity of their content.
Revised: We evaluated models on behavioral data from a Multiple Arrangement task where participants arranged NSD images according to the similarity of their content.

Original: This behavioral analysis of neuralnetwork models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capturehuman perceptual patterns and image semantics than early visual areas and this quantitative agreementincreases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’analysis discussed in Methods).
Revised: This behavioral analysis of neuralnetwork models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capturehuman perceptual patterns and image semantics than early visual areas and this quantitative agreementincreases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’analysis discussed in Methods).

Original: Importantly, responses patterns in their respective models groupsemantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region
Revised: Importantly, responses patterns in their respective models groupsemantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region

Original: VO).
Revised: VO).

Original: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes grouptogether in the left (Figure 5A), despite wide variations in the visual appearance of these categories.
Revised: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes grouptogether in the left (Figure 5A), despite wide variations in the visual appearance of these categories.

Original: This is remarkable, given that the models received no supervision from perceptual/semantic measures.
Revised: This is remarkable, given that the models received no supervision from perceptual/semantic measures.

Original: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Appendix Figure
Revised: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Appendix Figure

Original: A.10.
Revised: A.10.

Original: This suggests that these models, solely optimized for high-level visual cortical responseprediction – without any semantic supervision – can even serve as powerful models for uncoveringthe structure of mental representations.
Revised: This suggests that these models, solely optimized for high-level visual cortical responseprediction – without any semantic supervision – can even serve as powerful models for uncoveringthe structure of mental representations.

Original: Intriguingly, the representational structure of predictedresponses exhibits stronger agreement with perceived similarity than the representational structureof measured responses.
Revised: Intriguingly, the representational structure of predictedresponses exhibits stronger agreement with perceived similarity than the representational structureof measured responses.

Original: This suggests that both the perceptual and semantic information are likelymore explicit in predicted responses (even for held-out data) than measured responses. 
Revised: This suggests that both the perceptual and semantic information are likelymore explicit in predicted responses (even for held-out data) than measured responses. One possibleexplanation is

Revised: that

Original: One possibleexplanation is that fMRI measures of brain activity have undesired noise components and predictivemodels, in particular, response-optimized models as shown here, can denoise thisfMRI activity (seealso Appendix Figure A.3).(b)
Revised: fMRI measures of brain activity have undesired noise components and predictivemodels, in particular, response- optimized models as shown here, can denoise this

Revised: fMRI

Revised: activity (seealso Appendix Figure A.3).

Revised: (b) Monotonically increasing separability of category information along the ventral visual

Original: Monotonically increasing separability of category information along the ventral visualstream Task-optimized computational models have suggested that the ventral stream hierarchy‘untangles’ representations through a sequence of processing steps, so that representations of objectsand categories that are inseparable at the initial processing stage (e.g., V1) become untangled atthe last stage of the hierarchy [54, 23].
Revised: stream Task-optimized computational models have suggested that the ventral stream hierarchy ‘untangles’ representations through a sequence of processing steps, so that representations of objects

Revised: and

Revised: categories that are inseparable at the initial processing stage (e.g., V

Revised: become untangled at

Revised: the last stage of the hierarchy

Original: Here, we present a strong evidence for this increasedseparability through hypothesis-agnostic response-optimized models.
Revised: Here, we present a strong evidence for this increasedseparability through hypothesis-agnostic response-optimized models.

Original: Previous studies have shownthat the representational geometry for the same set of stimuli varies systematically across differentcortical areas, providing a useful signature for how information is transformed across differentcortical processing streams [54].
Revised: Previous studies have shownthat the representational geometry for the same set of stimuli varies systematically across differentcortical areas, providing a useful signature for how information is transformed across differentcortical processing streams [54].

Original: Here, we employ the representational similarity analysis (RSA)framework to relate emergent representations in response-optimized models of difference brainregions against a ground truth adjacency matrix defined by object category labels.
Revised: Here, we employ the representational similarity analysis (RSA)framework to relate emergent representations in response-optimized models of difference brainregions against a ground truth adjacency matrix defined by object category labels.

Original: Keeping model architecture the same, simply by changing the response targets for an encoding model from voxelsin visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extractedrepresentation.
Revised: Keeping modelarchitecture the same, simply by changing the response targets for an encoding model from voxelsin visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extracted representation.

Original: The increasingly prominent block-diagonal structure in the RSMs along the ventralvisual stream (Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlightsthat the distributed activity patterns to exemplars of the same category become increasingly moresimilar; and patterns to exemplars of different categories become progressively dissimilar alongthe processing stream, strongly supporting the computational goal of the ventral visual stream asa categorization system.
Revised: The increasingly prominent block-diagonal structure in the RSMs along the ventralvisual stream (Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlightsthat the distributed activity patterns to exemplars of the same category become increasingly moresimilar; and patterns to exemplars of different categories become progressively dissimilar alongthe processing stream, strongly supporting the computational goal of the ventral visual stream asa categorization system.

Original: We also quantified the separability of categorical information in differentvisual areas (Figure 5B).
Revised: We also quantified the separability of categorical information in differentvisual areas (Figure 5B).

Original: Here again, we see that this finding regarding increasing separability ofcategorical information holds across both datasets containing different visual categories.
Revised: Here again, we see that this finding regarding increasing separability ofcategorical information holds across both datasets containing different visual categories.

Original: Response-optimized models reveal high-level visual information within ventral-occipital cortex
Revised: Response-optimized models reveal high-level visual information within ventral-occipital cortex

Original: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model andwhether they are capable of discriminating between complex categories.
Revised: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model andwhether they are capable of discriminating between complex categories.

Original: Instead of looking at allmodel neurons emulating voxels, we built a compact neural representational space by passing a largestimulus set ( ∼ 27,000 images from the THINGS database [44]) through the predictive models andthen performing Principal Components Analysis (PCA) on the predicted responses of this network(Appendix A.6).
Revised: Instead of looking at allmodel neurons emulating voxels, we built a compact neural representational space by passing a largestimulus set ( ∼ 27,000 images from the THINGS database [44]) through the predictive models andthen performing Principal Components Analysis (PCA) on the predicted responses of this network(Appendix A.6).

Original: Visualizing the images from top concept categories that elicit highest and lowestactivation of individual principal components (PCs) reveals complex stimulus dimensions: the firstprincipal component, which explained ∼ 49% of the variance in responses strongly separates faces,animals and bodies from inanimate objects, despite vast variations in the visual appearance ofimages within sub-categories.
Revised: Visualizing the images from top concept categories that elicit highest and lowestactivation of individual principal components (PCs) reveals complex stimulus dimensions: the first PC, which explained ∼ 49% of the variance in responses strongly separates faces, animals andbodies from inanimate objects, despite vast variations in the visual appearance of images withinsub-categories.

Original: The second principal component, accounting for ∼ 16% variance,distinguished objects based on their shape (circular versus rectilinear objects), suggesting that suchshape features are important for VO.
Revised: The second PC, accounting for ∼ 16% variance, distinguished objects based on theirshape (circular vs. rectilinear), suggesting that such shape features are important for VO.

Original: Importantly, these features do not emerge when the sameanalysis is applied to early visual ROIs (V1-V4).
Revised: Importantly,these features do not emerge when the same analysis is applied to early visual ROIs (V1-V4).

Original: Response-optimized models exhibita shape bias A striking known discrepancy between human and machine vision is the shape vs. texture bias: human classification decisions are shape-biased whereas CNNstrained for CV tasks tend to classifyimages by texture [55, 56].
Revised: Response-optimized models exhibita shape bias A striking known discrepancy between human and machine vision is the shape vs. texture bias: human classification decisions are shape-biased whereas CNNstrained for CV tasks tend to classifyimages by texture [55, 56].

Original: We studywhether the shape bias of humans isreflected in response-optimized models of cortical regions that subservevisual categorization in humans.
Revised: We studywhether the shape bias of humans isreflected in response-optimized models of cortical regions that subservevisual categorization in humans.

Original: Wefirst train a linear layer on top ofthe predicted responses of responseoptimized models to map these responses to visual categories in the
Revised: Wefirst train a linear layer on top ofthe predicted responses of responseoptimized models to map these responses to visual categories in the

Original: ImageNet-16 database [55] using asmall subset of this database.
Revised: ImageNet-16 database [55] using asmall subset of this database.

Original: We thenassess the transfer performance ofresponse-optimized models for classification on held-out stimuli from the
Revised: We thenassess the transfer performance ofresponse-optimized models for classification on held-out stimuli from the

Original: ImageNet-16 dataset as well as the
Revised: ImageNet-16 dataset as well as the

Original: Silhouette dataset described in [55](Appendix A.10).
Revised: Silhouette dataset described in [55](Appendix A.10).

Original: We find that theclassification performance is significantly better than what would beexpected from the same network architecture with a random initialization (Figure 6B), suggesting that thebrain response-optimization objectiveyields effective representations.
Revised: We find that theclassification performance is significantly better than what would beexpected from the same network architecture with a random initialization (Figure 6B), suggesting that thebrain response-optimization objectiveyields effective representations.

Original: Further, this classification performance isbetter using models of intermediate and high-level visual areas (V4, LO and VO) than early visual ar- eas, even approaching the transfer performance of an MS-COCO-trained object classification networkwith the same architecture (Figure 6B,top, see Appendix A.10 for model details), providing furthersupport for the separability of categorical information in high-level, but not early visual cortex [11].
Revised: Further, this classification performance isbetter using models of intermediate and high-level visual areas (V4, LO and VO) than early visual areas, even approaching the transfer performance of an MS-COCO-trained object classification networkwith the same architecture (Figure 6B,top, see Appendix A.10 for model details), providing furthersupport for the separability of categorical information in high-level, but not early visual cortex [11].

Original: The LO model has the highest classification performance on the Silhouette dataset (which solelycontain shape information), consistent with and providing further evidence for previous hypothesesthat LO tracks object shape information [57].
Revised: The LO model has the highest classification performance on the Silhouette dataset (which solelycontain shape information), consistent with and providing further evidence for previous hypothesesthat LO tracks object shape information [57].

Original: We then evaluate the classification performance of thismapped response-optimized model on the diagnostic texture-shape cue conflict dataset [55] to assesswhether the model makes its categorization decisions based on shape or texture (Appendix A.10).
Revised: We then evaluate the classification performance of thismapped response-optimized model on the diagnostic texture-shape cue conflict dataset [55] to assesswhether the model makes its categorization decisions based on shape or texture (Appendix A.10).

Original: Wefollow [55] and define shape bias as the percentage of times the model classifies an image based onits shape over texture, provided it returned either the correct shape or correct texture label for theimage.
Revised: Wefollow [55] and define shape bias as the percentage of times the model classifies an image based onits shape over texture, provided it returned either the correct shape or correct texture label for theimage.

Original: A model is shape-biased if its shape bias > 50 % .
Revised: A model is shape-biased if its shape bias > 50 % .

Original: We observed that all response-optimizedmodels exhibit a shape bias (Figure 6B [bottom], Appendix Figure A.1), with this metric rangingfrom 56.7-69.4 % , suggesting that representations within the ventral visual stream are likely biasedtowards shapes than textures.
Revised: We observed that all response-optimizedmodels exhibit a shape bias (Figure 6B [bottom], Appendix Figure A.1), with this metric rangingfrom 56.7-69.4 % , suggesting that representations within the ventral visual stream are likely biasedtowards shapes than textures.

Original: This is in stark contrast to standard CV models which show a texturebias, for e.g., the same architecture trained for multi-label object classification on MS-COCO shows astrong texture bias and performs significantly worse on the Silhouettes dataset (Figure 6B, gray bar),suggesting that the distribution of NSD stimuli (which are drawn from MS-COCO) is not sufficientto drive the shape bias and the training objective (matching to neural responses rather than objectcategory labels) matters.
Revised: This is in stark contrast to standard CV models which show a texturebias, for e.g., the same architecture trained for multi-label object classification on MS-COCO shows astrong texture bias and performs significantly worse on the Silhouettes dataset (Figure 6B, gray bar),suggesting that the distribution of NSD stimuli (which are drawn from MS-COCO) is not sufficientto drive the shape bias and the training objective (matching to neural responses rather than objectcategory labels) matters.

Original: Further, as shown in Figure 6B (bottom), it is not the linear mapping toneural response but the way features are optimized that explains the differing shape/texture bias oftask-optimized and response-optimized encoding models.
Revised: Further, as shown in Figure 6B (bottom), it is not the linear mapping toneural response but the way features are optimized that explains the differing shape/texture bias oftask-optimized and response-optimized encoding models.

Original: We next performed a feature visualizationanalysis to find visual patterns thatwould maximally activate individual model neurons emulatingbrain voxels indifferent response-optimized encoding models.
Revised: We next performed a feature visualizationanalysis to find visual patterns that maximally activate individual model neurons emulating brainvoxels in response-optimized encoding models.

Original: Following [58], we start with arandom noise input x 0 for each model neuron i and iteratively update the input along the gradient∂a i ∂x to synthesize inputs that would result in higher and higher predicted activation a i for that neuron(see Suppl.
Revised: Following [58], we start with a random noise inputx 0 for each model neuron i and iteratively update the input along the gradient ∂a i ∂x to synthesizeinputs that would result in higher and higher predicted activation a i for that neuron (Appendix A.9).

Original: Methods). Visual inspection shows that the features increase in complexity as we go fromearly to high-level visual ROI models: simple edge-like features or high-low frequency detectors areseen in V1, curves and edges are seen in V2, more complex curvatures in V4 and finally, fully-formedcomplex shapes like concentric hexagons are seen in LO and VO (Figure 6A, Appendix Figure A.7).
Revised: Visual inspection shows that the features increase in complexity as we go from early to high-levelvisual ROI models: simple edge-like features or high-low frequency detectors are seen in V1, curvesand edges are seen in V2, more complex curvatures in V4 and finally, fully-formed complex shapeslike concentric hexagons are seen in LO and VO (Fig. 6A, Appendix Fig. A.7).

Original: We also validated this increased complexity using quantitative measures based on compression ratios,as shown in Appendix Table A.3.
Revised: We also validated thisincreased complexity using quantitative measures based on compression ratios (Appendix Table A.3).

Original:  Discussion
Revised:  Discussion

Original: In this study, we leveraged the large sample size and natural variation represented in the Natural
Revised: In this study, we leveraged the large sample size and natural variation represented in the Natural

Original: Scenes Dataset to train deep neural networks directly on the brain response.
Revised: Scenes Dataset to train deep neural networks directly on the brain response.

Revised: Concurrently, recentpapers [31, 30] have also demonstrated that when trained with large amounts of data such as NSD,the response-optimization approach can perform competitively with state-of-the-art task-optimizedmodels (discussed in more detail in Appendix C.1).

Original: We studied modelgeneralization in various settings: from synthetic stimuli to novel fMRI datasets with differentpreprocessing.
Revised: We studied model generalization in varioussettings: from synthetic stimuli to novel fMRI datasets with different preprocessing.

Original: Our results recapitulated key features of early visual areas (e.g. their precise retinotopicorganization), representational properties in higher level areas.
Revised: Our resultsrecapitulated key features of early visual areas (e.g. their precise retinotopic organization) andrepresentational properties in higher level areas.

Original: We also demonstrated that responseoptimized models can capture human behavior, including not only object categorization, but also otherfine-grained measures such as perceived image similarity.
Revised: We also demonstrated that response-optimizedmodels can capture human behavior, including not only object categorization, but also other finegrained measures such as perceived image similarity.

Original: The response-optimization approach, inprinciple, allows exploring infinite paths in the model space, and when optimized properly, convergingtowards one that explains maximum possible variance in the data.
Revised: The response-optimization approach, inprinciple, allows exploring infinite paths in the model space, and when optimized properly, convergingtowards one that explains maximum possible variance in the data.

Original: Since any features that emergein these optimized networks are a result of optimization to match neural targets, unconfounded byany other supervision signal, these models can provide stronger tests about the visual representationswithin different visual areas.
Revised: Since any features that emergein these optimized networks are a result of optimization to match neural targets, unconfounded byany other supervision signal, these models can provide stronger tests about the visual representationswithin different visual areas.

Original: In terms of human object categorization behavior, our models lack significantly compared to
Revised: In terms of human object categorization behavior, our models lack significantly compared to

Original: ImageNet-supervised DNN models.
Revised: ImageNet-supervised DNN models.

Original: While part of this may be due to the domain shift, the restricted transfer learning setting (a single linear layer mapping), small size of the transfer set orsimply the coarse spatial resolution of fMRI, this gap is worth exploring.
Revised: While part of this may be due to the domain shift, the restricted transfer learning setting (a single linear layer mapping), small size of the transfer set orsimply the coarse spatial resolution of fMRI, this gap is worth exploring.

Original: Further, using neural data tobuild better models for other communities (e.g. computer vision) is another scientifically fascinatingdirection and can develop reinforcing cycles.
Revised: Further, using neural data tobuild better models for other communities (e.g. computer vision) is another scientifically fascinatingdirection and can develop reinforcing cycles.

Original: Studies have shown that explicitly encouraging neuralnetworks to build representations similar V1 representations can improve their robustness undernoise and adversarial attacks [59, 60].
Revised: Studies have shown that explicitly encouraging neuralnetworks to build representations similar V1 representations can improve their robustness undernoise and adversarial attacks [59, 60].

Original: In a similar vein, regularization with high-level visual corticalactivity might help grant models brain-like biases (e.g. shape bias) and close the gap between humanand machine vision.
Revised: In a similar vein, regularization with high-level visual corticalactivity might help grant models brain-like biases (e.g. shape bias) and close the gap between humanand machine vision.

Revised: Alternatively, probing the connections in response-optimized models, whichalready exhibit a shape bias, using model pruning and attribution methods can yield a mechanisticunderstanding of the shape bias in neural activity and reveal principles that may enable CV modelsto attain similar representations.

Revised: Recent studies have inspired optimism that studying connectionweights to reveal underlying computational principles may be tractable [33, 61].

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] They are described within the
Revised: [Yes] They are described within the

Original: Discussion section
Revised: Discussion section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We don’t
Revised: [N/A] We don’t

Original: foresee any negative societal impacts of this work.
Revised: foresee any negative societal impacts of this work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes] I’ve read all guidelines
Revised: [Yes] I’ve read all guidelines

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical
Revised: No theoretical

Original: results are discussed in this paper.
Revised: results are discussed in this paper.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical results
Revised: No theoretical results

Original: are discussed in this paper.
Revised: are discussed in this paper.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Yes, the code is
Revised: [Yes] Yes, the code is

Original: included in the Supplementary Material.
Revised: included in the Supplementary Material.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were

Original: chosen)?
Revised: chosen)?

Original: [Yes] Yes, all training details are included in the Methdos and Supplementary
Revised: [Yes] Yes, all training details are included in the Methdos and Supplementary

Original: Methods.
Revised: Methods.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Revised: [Table]

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] Yes, they are displayed in all figures.
Revised: [Yes] Yes, they are displayed in all figures.

Original:  (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] Yes.
Revised: [Yes] Yes.

Original: All models were trained, validated and analysed on an NVIDIA Titan Xp GPU.
Revised: All models were trained,validated and analysed on an NVIDIA Titan Xp GPU.

Original: Training each response-optimized model on 1 GPU took ∼ 8 hours.
Revised: Training each response-optimizedmodel on 1 GPU took ∼ 8 hours.

Original: Additional details are listed in the Supplementary.
Revised: Additional details are listed in the Supplementary.4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] e.g. [47, 61]
Revised: [Yes] e.g. [47, 62]

Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] The assets employed in this study were 
Revised: [No] The assets employed in this study were [Table]

Original: made publicly available by the creators
Revised: made publicly available by the creators

Original:  (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Table] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original:  (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [No] The assets employed in this study were made publicly available by the creators 
Revised: [No] The assets employed in this study were made publicly availableby the creators [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: [No] The assets don’t contain any offensive content to our knowledge. 
Revised: [No] The assets don’t contain any offensive contentto our knowledge. [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A](c)

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]