Original: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models
Revised: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Decades of experimental research based on simple, abstract stimuli has revealedthe coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complexvisual categories in the anterior ventral stream.
Revised: Decades of experimental research based on simple, abstract stimuli has revealedthe coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complexvisual categories in the anterior ventral stream.

Original: However, these studies are, byconstruction, constrained by their a priori hypotheses.
Revised: However, these studies are, byconstruction, constrained by their a priori hypotheses.

Original: Furthermore, beyond theearly stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.
Revised: Furthermore, beyond theearly stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.

Original: In this work, wepropose to employ response-optimized encoding models trained solely to predictthe functional MRI activation, in order to gain insights into the tuning propertiesand representational transformations in the series of areas along the ventral visualpathway.
Revised: In this work, wepropose to employ response-optimized encoding models trained solely to predictthe functional MRI activation, in order to gain insights into the tuning propertiesand representational transformations in the series of areas along the ventral visualpathway.

Original: We demonstrate the strong generalization abilities of these models onartificial stimuli and novel datasets.
Revised: We demonstrate the strong generalization abilities of these models onartificial stimuli and novel datasets.

Original: Intriguingly, we find that response-optimizedmodels trained towards the ventral-occipital and lateral-occipital areas, but notearly visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.
Revised: Intriguingly, we find that response-optimizedmodels trained towards the ventral-occipital and lateral-occipital areas, but notearly visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.

Original: We further probe the trainednetworks to reveal representational biases in different visual areas and generateexperimentally testable hypotheses.
Revised: We further probe the trainednetworks to reveal representational biases in different visual areas and generateexperimentally testable hypotheses.

Revised: Our analyses suggest a shape-based processingalong the ventral visual stream and provide a unified picture of multiple neuralphenomena characterized over the last decades with controlled fMRI studies.

Original:  Introduction
Revised:  Introduction

Original: Understanding the nature of representations and computations in the visual system has been alongstanding goal in neuroscience.
Revised: Understanding the nature of representations and computations in the visual system has been alongstanding goal in neuroscience.

Original: Substantial progress been made  by presenting model organismswith simple, abstract stimuli like noise or sine-wave gratings and studying the evoked responses.
Revised: Substantial progress been made in this quest for early stages ofthe visual cortex by presenting model organisms with simple, abstract stimuli like noise or sine-wavegratings and studying the evoked responses, such as the peak of their tuning curves.

Original: These carefully crafted experiments, for instance, revealed the presence of edge detectors in theprimary visual cortex [1] and sparked the search for the ‘optimal input’ beyond early visual areasacross the visual cortical hierarchy.
Revised: These carefullycrafted experiments, for instance, revealed the presence of edge detectors in the primary visualcortex [1] and sparked the search for the ‘optimal input’ beyond early visual areas across the visualcortical hierarchy.

Original: Significant efforts have also been made to understand the featurecomplexity of tuning in V4 and IT neurons [2, 3, 4, 5].

Original: Over the last two decades, through numerousexperiments, recording and analysis techniques, a conceptual understanding has emerged that earlyvisual areas extract low-level features like edges or curves, mid-level regions extract complex localshapes, and high-level regions encode semantic categories, like faces or scenes [6, 7, 8, 9, 10].
Revised: Over the last two decades, through numerous experiments, recording and analysistechniques, a conceptual understanding has emerged that early visual areas extract low-level featureslike edges or curves, mid-level regions extract complex local shapes, and high-level regions encodeand represent different semantic categories, like faces or scenes [2, 3, 4, 5, 6].

Original: Whilemuch of the details remain unknown, there is accumulating evidence that visual categorization occursin the ventral temporal cortex in humans [11].
Revised: While the underlyingrepresentational transformations that enable flexible and efficient visual perception remain unknown,there is accumulating evidence that visual categorization occurs in the ventral temporal cortex inhumans [7].

Original: This understanding  was enabled by synthesizingfindings across different hypothesis-driven  studies, most of them employing artificial stimuli atypicalof real-world vision, thus lacking ethological validity.
Revised: This understanding of the visual system was enabled by synthesizing findings acrossseveral different hypothesis-driven experimental studies, most of them employing artificial stimuliatypical of real-word vision, thus lacking ethological validity.

Original: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks,have emerged as the most accurate models of the primate ventral visual stream [12, 13, 14, 15].
Revised: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks, haveemerged as the most accurate models of the primate ventral visual stream [8].

Original: Furthermore, these CV task-optimized models offer a hierarchical correspondence to visual areasalong the ventral stream: early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [12, 16, 17, 18, 19, 20, 21, 22].
Revised: Furthermore, these CVtask-optimized models offer a hierarchical correspondence to visual areas along the ventral stream:early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [8, 9].

Original: They provide intelligibility in the formof the goal-driven perspective, highlighting the functional role of representations [23].
Revised: They provide intelligibility in the form of the goal-driven perspective, highlighting the functionalrole of representations [10].

Original: However,we are far from a complete understanding of the top-down computational objectives (the batteryof ethologically-relevant tasks) that may have shaped neural representations through evolution orexperience-based learning; in the absence of right hypotheses, the task-optimized approach can fallshort.
Revised: However, we are far from a complete understanding of the top-downcomputational objectives (the battery of ethologically-relevant tasks) that may have shaped neuralrepresentations through evolution or experience-based learning; in the absence of right hypotheses,the task-optimized approach can fall short.

Original: Studies have already revealed significant inconsistencies between these models and humanbehavior and gaps remain in their respective robustness to distortions, image-level consistencyetc [24].

Original: Recently, the idea of using neural data in the model development process has gainedsome traction, and demonstrated success in increasing both model-human behavior [25, 26, 27],and model-brain alignment, while revealing novel insights about information processing in neuralsystems [28, 29, 30, 31].

Original: In this study, we leverage recent advances in large-scale fMRI data collection [32] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.
Revised: In this study, we leverage recent advances in large-scale fMRI data collection [11] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.

Original: Importantly,the stimulus set employed in this study contains crowded images of multiple  objects in their natural contexts, thus being more typical of everyday scenes and allowing us to characterize neuralrepresentations and computations in rich, naturalistic conditions.
Revised: Importantly,the stimulus set employed in this study contains crowded images of multiple common objects in theirnatural contexts, thus being more typical of everyday scenes and allowing us to characterize neuralrepresentations and computations in rich, naturalistic conditions.

Original: We demonstrate the remarkableability of these “response-optimized” neural encoding models in reproducing the stimulus-responserelationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassingthe unprecedented prediction performance of task-optimized models.
Revised: We demonstrate the remarkableability of these “response-optimized” neural encoding models in reproducing the stimulus-responserelationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassingthe unprecedented prediction performance of task-optimized models.

Original: We show that these modelsgeneralize well to contexts that are vastly different from their training conditions (artificial stimuli,novel subjects, etc.), providing a strong quantitative validation for the response-optimization approachof building encoding models for neurobiological visual systems beyond the retina [33].
Revised: We show that these modelsgeneralize well to contexts that are vastly different from their training conditions (artificial stimuli,novel subjects, etc.), providing a strong quantitative validation for the response-optimization approachof building encoding models for neurobiological visual systems beyond the retina [12].

Original: Our approachfurther goes beyond prediction accuracy and aims to gain insights about neuronal tuning properties.
Revised: Our approachin this paper, however, goes beyond prediction accuracy and aims to gain insights about neuronaltuning properties.

Original: Specifically, we implement neural encoding models that disentangle the processing of the “what” andthe “where.”
Revised: Specifically, we implement neural encoding models that disentangle the processingof the “what” and the “where.”

Original: This allows us to reveal the selectivity of individual brain voxels along both locationand appearance dimensions.
Revised: This allows us to reveal the selectivity of individual brain voxels alongboth location and appearance dimensions.

Original: We first demonstrate that the estimated spatial topographic organizationfrom our neural encoding models exhibits remarkable agreement with the fine-scaled retinotopicorganization of the cortex, which is typically measured with dedicated, painstaking experimentalparadigms.
Revised: We first demonstrate that the estimated spatial topographicorganization from our neural encoding models exhibits remarkable agreement with the fine-scaledretinotopic organization of the cortex, which is typically measured with dedicated, painstakingexperimental paradigms.

Original: We analyze the features that emerge in the neural encoding models to understand the tuning properties of cortical voxels.
Revised: We analyze the features that emerge in the neural encoding models tounderstand the tuning properties of cortical voxels.

Original: This analysis suggests shape-biased tuning across the ventral visualstream, increasing in complexity from Gabor-like features in V1, curves/angles in V2 to concentricand complex shape processing in V4 and higher-order lateral-occipital and ventral-occipital ROIsrespectively.
Revised: This analysis suggests shape-biased tuning acrossthe ventral visual stream, increasing in complexity from Gabor-like features in V1, curves and anglesin V2 to concentric (circular) and complex shape processing in V4 and higher-order lateral-occipitaland ventral-occipital ROIs respectively.

Original: We also go beyond single-voxel analysis to characterize how representations evolvealong the ventral visual pathway.
Revised: We also go beyond single-voxel analysis to characterizehow representations evolve along the ventral visual pathway.

Original: Probing the information encoded in model-predicted patterns ofactivity reveals the monotonically increasing separability of category information along the ventralstream hierarchy, strongly supporting the computational goal of the ventral visual stream as a visualcategorization system.
Revised: Probing the information encoded inmodel-predicted patterns of activity reveals the monotonically increasing separability of categoryinformation along the ventral stream hierarchy, strongly supporting the computational goal of theventral visual stream as a visual categorization system.

Original: We further demonstrate the correspondence between model representations (ofhigh-level visual areas) and mental representations (as assessed with human similarity judgements),suggesting that these response-optimized models inherently capture human psychological judgements,without any explicit supervision to do so.
Revised: We further demonstrate the correspondencebetween model representations (of high-level visual areas) and mental representations (as assessedwith human similarity judgements), suggesting that these response-optimized models inherentlycapture human psychological judgements, without any explicit supervision to do so.

Original: Furthermore, a principle components analysis reveals thatmodels of high-level visual ROIs capture complex stimulus dimensions like faces, bodies and shapes.
Revised: Furthermore, aprinciple components analysis reveals that models of high-level visual ROIs capture complex stimulusdimensions like faces, bodies and shapes.

Original: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, naturalistic datasetscan be used to confirm neural phenomena characterized painstakingly using controlled experimentalparadigms that might be hard to replicate in individual subjects.
Revised: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, yet naturalisticdatasets can be used to confirm neural phenomena characterized painstakingly using controlledexperimental paradigms that might be hard to replicate in individual subjects.

Original: More importantly, however, we showhow in silico experiments on neural encoding models can generate novel hypotheses and provide apowerful framework for building broad, generalizable theories about neural information processing.
Revised: More importantly,however, we show how in silico experiments on neural encoding models can generate novel hypothesesand provide a powerful framework for building broad, generalizable theories about neural informationprocessing.

Original:  Materials and Methods
Revised:  Materials and Methods

Original: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is providedelsewhere [32] (see also the Appendix).
Revised: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is providedelsewhere [11] (see also Supplemental Methods).

Original: The  dataset contains measurements of fMRI responses from8 participants who each viewed 9,000–10,000 natural scenes.
Revised: The NSD dataset contains measurements of fMRIresponses from 8 participants who each viewed 9,000–10,000 natural scenes (22,000–30,000 trials)over the course of 30–40 sessions.

Original: A special set of 1,000 images wereshared across subjects; the remaining were mutually exclusive.
Revised: A special set of 1,000 images were shared across subjects; theremaining were mutually exclusive across subjects.

Original: We focused on modeling responseswithin 5 visual cortical ROIs in the study.
Revised: We focused on modeling responses within 5 visualcortical ROIs in the study.

Original: Three ROIs belonging to the retinoptic early visual cortex,namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session [32].
Revised: Three ROIs belonging to the retinoptic early visual cortex, namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session for each subject [11].

Original: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipitalareas (LO1-2) were delineated using a visual probabilistic atlas [34](Figure 1B).
Revised: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipitalareas (LO1-2) were delineated using a visual probabilistic atlas [13](Figure 1B)

Original: Response-optimized neural encoding model
Revised: Response-optimized neural encoding model

Original: We trained separate voxel-level neural encodingmodels for each of the above regions with the same backbone architecture.
Revised: We trained separate voxel-level neural encodingmodels for each of the above regions with the same backbone architecture.

Original: Keeping the architectureconstant across visual areas ensured that differences in emergent representations are driven mainlyby the nature of response patterns.
Revised: Keeping the architectureconstant across visual areas ensured that differences in emergent representations are driven mainlyby the nature of response patterns.

Original: The predictive model comprises a convolutional neural network(CNN) feature extractor shared across all subjects, but unique to specific visual areas.
Revised: The predictive model comprises a convolutional neural network(CNN) feature extractor shared across all subjects but unique to specific visual areas.

Original: We employ alinear readout model on top of the feature space to predict the responses of individual voxels in aspecific region.
Revised: We employ alinear readout model on top of the feature space to predict the responses of individual voxels in aspecific region.

Original: The linear readout is factorized into spatial and feature dimensions following [35](Figure 1A).
Revised: The linear readout is factorized into spatial and feature dimensions following [14](Figure 1A).

Original: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).
Revised: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).

Original: The shared CNN feature extractor consists of four  convolutional blocks, with each block comprisingthe following feedforward computations: two 3 × 3 convolutional layers, each followed by an innerbatch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.
Revised: The shared CNN feature extractor consists of four sequential convolutional blocks, with each blockcomprising the following feedforward computations: two 3 × 3 convolutional layers, each followedby an inner batch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.

Original: We employ E(2)-steerable convolutions in all our models to extract features invariant to orientation [36].
Revised: Instead of regular convolutions, we employ E(2)-steerable convolutions in all our models to extractfeatures invariant to orientation [15].

Original: Thismodeling choice is inspired by neural computations in early visual areas where groups of neurons areknown to perform similar computations, e.g., edge or curve detection, at different orientations.
Revised: This modeling choice is inspired by neural computations inearly visual areas where groups of neurons are known to perform similar computations, e.g., edge orcurve detection, at different orientations.

Original: Eachconvolutional layer contains 48 feature sets extracted at 8 orientations.
Revised: Each convolutional layer contains 48 feature sets extractedat 8 orientations.

Original: Weights of the readout are acomputed as outer products between a spatial filter and a feature vector.
Revised: Weights of the readout are a computed as outer products between a spatial filter anda feature vector.

Original: The spatial filter further hasa positivity constraint (enforced using rectification) and is length-normalized for each voxel.
Revised: The spatial filter further has a positivity constraint (enforced using rectification) andis length-normalized for each voxel.

Original: Training and testing response-optimized encoding models We used 4 NSD subjects for trainingand reserved the other 4 subjects for studying generalization.
Revised: Training and testing response-optimized encoding models We used 4 NSD subjects for trainingand reserved the other 4 subjects for studying generalization.

Original: The first group collectively saw 37,000natural scene images, including the 1,000 shared images.
Revised: The 4 training subjects collectivelysaw 37,000 natural scene images, including the 1,000 shared images.

Original: We used the 1,000 shared images for testingour models and split the remaining stimulus set into 35,000 training and 2,000 validation images.
Revised: We used the 1,000 sharedimages for testing our models and split the remaining stimulus set into 35,000 training and 2,000validation images.

Original: All parameters of the neural encoding model were optimized jointly to minimize the masked meansquared error between the predicted and measured fMRI response.
Revised: All parameters of the neural encoding model were optimized jointly to minimizethe masked mean squared error between the predicted and measured fMRI response.

Original: This loss allows us to propagateerrors through the shared CNN even if the subjects are not exposed to common stimuli since we canexclude the subjects/voxels for which the target response is not present from the loss calculation.
Revised: This loss allowsus to propagate errors through the shared CNN even if the subjects are not exposed to commonstimuli since we can exclude the subjects/voxels for which the target response is not present from theloss calculation.

Original: Wequantify ‘predictive accuracy’ on the test images as the Pearson’s correlation coefficient between thepredicted and measured fMRI response at each voxel.
Revised: We quantify ‘predictive accuracy’ on the test images as the Pearson’s correlationcoefficient between the predicted and measured fMRI response at each voxel.

Original: We also compare this against the noise-ceiling,expressed as the signal-to-noise ratio when considering the response variability of each voxel across3 repetitions per image (see Appendix A.5).
Revised: We also compare thisagainst the noise-ceiling (see Suppl.

Original: Table A.1 provides a summary of all analyses methodsand baselines used for evaluating and interpreting the proposed response-optimized models.
Revised: Methods for further details).

Original: Baseline models (a) Category ideal observer model: We fit a non-image-computable modelusing human-generated annotations for NSD images (obtained from the MS-COCO database). Theinput to  the categorical model is a k-hot encoding vector corresponding to the 80 object categoriesannotated in  the database, where each element indicates the presence/absence of the correspondingcategory in the image.
Revised: Generalization to new subjects We tested the generalization of response-optimized models to 4test subjects in NSD by freezing the weights of the shared CNN and only training linear readouts. Wevaried the size of the training set for optimizing linear readout weights from a mere 100 stimulusresponse pairs to a large set of 4,500 pairs from novel subjects (see Suppl. Methods).

Original: We fitted l 2 regularized linear regression models on this space to predictvoxel responses for held-out stimuli (Appendix A.8). (b) Task-optimized Models: We investigatedseveral DNN models, including AlexNet [37], ResNet-50 [38], DenseNet [39] and CORnet-s [40],optimized for object recognition on the large-scale ImageNet dataset [41] (Appendix A.3, Table
Revised: early visual areas The factorized readout allow us to separatespatial and feature selectivity. We assessed whether the spatial selectivity of our model trained onnatural stimuli agrees with measurements from carefully controlled experiments that use artificialstimuli.

Original: A.2).
Revised: The pRF is defined as the region of the visual field within which a stimulus results inincreased aggregated activity across a population of neurons, as reflected in fMRI.

Original: Since these models can demonstrate lower prediction performance due to the domain shiftbetween ImageNet and NSD stimuli, we also employed a ResNet-50 architecture pre-trained forobject detection on the MS-COCO dataset [42] as an additional baseline (since NSD stimuli weredrawn from the MS-COCO dataset).
Revised: It is commonlyestimated in a retinotopy experiment using pRF localizer scans. Based on the analysis of thesedata, each brain voxel’s retinotopic organization can then be described with three parameters:

Original: For CORnet-S, we pre-select the layer homologous to eachvisual region and employ a spatial × feature factorized readout (same as the response-optimizedmodel) on top of the fixed feature backbone, allowing a more direct comparison to the proposed model. For the remaining models, we followed the more standard procedure for fitting task-optimizedmodels to voxel responses based on a full linear readout.
Revised: thepRF location in polar coordinates (polar angle and eccentricity), and size. Using the voxel-wisespatial mask of the readout in our encoding model, we estimated the pRF location as the average ofthe coordinates weighted by the learned spatial mask value at each grid location.

Original: We extracted features from pre-trainedlayers of each network and fitted l 2 regularized linear regression models separately for each layer. The regularization parameter was optimized independently for each subject, each layer, each modeland for voxels in each visual area by testing among 8 log-spaced values in [1e-4, 1e4]. For each visualarea and model, we identified the layer that best predicts responses on the validation set.
Revised: To compare theestimated eccentricity against the corresponding measurements from pRF localizers, henceforth called‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient between the measuredand predicted eccentricity arrays in each ROI. To compare the estimated polar angle values againstcorresponding measurements from pRF localizer scans, henceforth called ‘polar angle agreement’ ,we adopted a measure from circular statistics called circular correlation coefficient [18], due to thecircular nature of angular data (see Supp.

Original: These modellayers were used to quantify test performance.
Revised: Methods).

Original: (c) Response-optimized rotation symmetries): Here, we train response-optimized models with the same architecture as the proposed models butwithout sharing weights across filter orientations. Characterizing neural representations by behavioral comparisons: (a) Assessing separabilityof category and semantic information along the visual hierarchy To probe the learned featuraltuning in the response-optimized model for each visual area, we used representational similarityanalysis (RSA) [43].
Revised: Characterizing neural representations by behavioral comparisons: (a) Assessing separabilityof category and semantic information along the visual hierarchy In addition to probing thelearned readout layer that gives insights into spatial tuning properties of voxels, we were interested inthe analysis of the response-optimized feature extractor for each visual area to understand featuraltuning. To achieve this, we used representational similarity analysis (RSA) [19].

Original: Here, we correlated the patterns of predicted activity between multipleexemplars from different categories of stimuli to obtain a representational similarity matrix (RSM)for each visual area.
Revised: Here, we correlatedthe patterns of predicted activity between multiple exemplars from different categories of stimuli toobtain a representational similarity matrix (RSM) for each visual area.

Original: Further, we quantified the separability of categorical information within thesesimilarity matrices by computing a correlation coefficient (Kendall’s τ ) between model RSMs againsta ground truth adjacency matrix defined by object category labels.
Revised: Further, we quantified theseparability of categorical information within these similarity matrices by computing a correlationcoefficient (Kendall’s τ ) between model RSMs against a ground truth adjacency matrix definedby object category labels.

Original: Specifically, the elements of thismatrix are 1 if the corresponding two images belong to different categories and 0 if the imagesbelong to the same category.
Revised: Specifically, the elements of this matrix are 1 if the correspondingtwo images belong to different categories and 0 if the images belong to the same category.

Original: This analysis was first performed using a subset of 104 images fromthe THINGS database [44] that belong to a pre-defined set of categories for ease of visualization.
Revised: Thisanalysis was first performed using a subset of 104 images from the THINGS database [20] thatbelong to a pre-defined set of categories for ease of visualization.

Original: These categories were chosen in accordance with previous fMRI studies employing RSA [45, 46]and include the following: { face, hand, elephant, cat, plant, fruit, car, tool } .
Revised: These categories were chosen inaccordance with previous fMRI studies employing RSA [21, 22] and include the following: { face,hand, elephant, cat, plant, fruit, car, tool } .

Original: Subsequently, we alsovalidated the obtained results using another large-scale vision dataset, namely ImageNet-16 [47].
Revised: Subsequently, we also validated the obtained results usinganother large-scale vision dataset, namely ImageNet-16 [23].

Original: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images andcomputed the similarity between model RSMs and the categorical adjacency matrix following thesame approach described for the THINGS dataset.
Revised: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images and computed the similarity betweenmodel RSMs and the categorical adjacency matrix following the same approach described for the THINGS dataset.

Original: To assess visual information flow in relation tohigh-level semantic understanding, we computed pairwise similarity between predicted responses forheld-out NSD images to construct model-predicted RSMs for each visual area.
Revised: To assess visual information flow in relation to high-level semantic understanding,we computed pairwise similarity between predicted responses for held-out NSD images to construct model-predicted RSMs for each visual area.

Original: We constructed asemantic RSM by correlating the 80-D binary category vectors (which indicate whether each categorywas present in the image or not) for every pair of these NSD images and quantified agreement withmodel-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).
Revised: We constructed a semantic RSM by correlating the80-D binary category vectors (which indicate whether each category was present in the image or not)for every pair of these NSD images and quantified agreement with model-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).

Original: Wecomputed this agreement using measured RSMs for all visual areas as well.
Revised: We computed this agreement usingmeasured RSMs for all visual areas as well.

Original: In RSA described hereand anywhere else, error bars (95 % CI) are computed by performing bootstrap sampling over voxels.
Revised: In RSA described here and anywhere else, error bars(95 % CI) are computed by performing bootstrap sampling over voxels.

Original: (b) Assessing alignment with human perception We compared model representations againstbehavioral data from a Multiple Arrangement task where participants interpreted 100 NSD imagesand arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ wasreleased with NSD [32]).
Revised: Assessing alignment with human perception We compared model representations againstbehavioral data from a Multiple Arrangement task where participants interpreted 100 NSD imagesand arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ wasreleased with NSD [11]).

Original: Critically, these images were not seen by any of the response-optimized models.
Revised: Critically, these images were not seen by any of the response-optimizedmodels.

Original: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROIusing the correlation distance metric for quantifying pairwise dissimilarities between model-predictedvoxel responses.
Revised: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROIusing the correlation distance metric for quantifying pairwise dissimilarities between model-predictedvoxel responses.

Original: We compared model RDMs against human pairwise dissimilarity judgements usingthe Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.
Revised: We compared model RDMs against human pairwise dissimilarity judgements usingthe Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.

Original: Results
Revised: Characterizing shape bias of model representations We first train a linear layer on top of thepredicted responses of response-optimized models to map these responses to visual categories inthe ImageNet-16 database using a small subset of this database. Next, we evaluate the classificationperformance of these linear classifiers on held-out images from the ImageNet-16 dataset and Silhouettes dataset [24]. Finally, we quantify the shape bias of all models using 1,200 images from thetexture-shape cue conflict dataset [24] (see Suppl. Methods). Characterizing voxel tuning through feature visualization We performed a feature visualizationanalysis to find visual patterns that would maximally activate individual model neurons emulatingbrain voxels in different response-optimized encoding models.

Revised: Following [25], we start with arandom noise input x 0 for each model neuron i and iteratively update the input along the gradient∂a i ∂x to synthesize inputs that would result in higher and higher predicted activation a i for that neuron(see Suppl.

Original:  Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B, Appendix Figure A.2, A.6).
Revised: Methods).  Results Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B).

Original: Next, we investigated the possible quantitative advantages of learninga rotationally-symmetric feature space against the representations from a standard convolutionalarchitecture with no weight sharing across orientations.
Revised: Next, we investigated the possible quantitative advantages of learning a rotationally-symmetric featurespace against the representations from a standard convolutional architecture with no weight sharingacross orientations.

Original: Importantly, we observe that the proposedrotationally-symmetric convolutional architecture performs on par with a standard CNN (Figure 2A),even outperforming the latter in certain visual ROIs, despite containing fewer parameters.
Revised: Importantly, we observe that the proposed rotationally-symmetric convolutionalarchitecture performs on par with a standard CNN (Figure 2A), even outperforming the latter incertain visual ROIs, despite containing fewer parameters.

Original: Thissuggests that sharing weights across filter orientations provides a strong inductive bias, allowing usto fit expressive models efficiently.
Revised: This suggests that sharing weights acrossfilter orientations provides a strong inductive bias, allowing us to fit expressive models efficiently.

Original: We further compared these models against ideal observer andtask-optimized models, and found that response-optimized models consistently outperform thesebaselines in early visual areas.
Revised: Wefurther compared these models against semantic and task-optimized models, and found that responseoptimized models consistently outperform these baselines in early visual areas.

Original: In high-level visual areas, the response-optimized modelsignificantly outperforms the ideal observer model, while performing on par with or even surpassing sometask-optimized models.
Revised: In high-level visualareas, the response-optimized model outperforms the semantic model, while performing on par withtask-optimized model.

Original: Next, we assessed how each of these predictivemodels generalizes to the remaining set of 4 testsubjects.
Revised: Next, we assessed how each of these predictive models generalizes to the remaining set of 4 testsubjects.

Original: This analysis intended to compare thestatistical efficiency afforded by different representations [48].
Revised: This analysis intended to compare the statistical efficiency afforded by different representations [26].

Original: As such, we examined the trendof how much stimulus-response data the models need to achieve a certain level of predictionperformance [48].
Revised: As such, we examined the trend of how much stimulus-response data the models need toachieve a certain level of prediction performance [26].

Original: Here, we freeze the sharedfeature extractor network and vary the amountof stimulus-response pairs from the 4 new subjects to train the linear readouts (Appendix A.7).
Revised: Here, we freeze the shared feature extractornetwork and vary the amount of stimulus-response pairs from the 4 new subjects to train the linearreadouts.

Original: We do the same for the task-optimized baselines,where the linear prediction model is fit on newsubjects.
Revised: We do the same for the task-optimized baseline, where the linear prediction model is fiton new subjects.

Original: As shown in Figure 2, the responseoptimized models can achieve a significantlyhigh accuracy even when only just 5 % of thetraining examples from novel subjects ( ∼ 500samples) are used to train the readout.
Revised: As shown in Figure 2, the response-optimized models can achieve a significantlyhigh accuracy even when only just 5 % of the training examples from novel subjects ( ∼ 500 samples)are used to train the readout.

Original: This remarkable generalization of response-optimizednetworks, exceeding that of task-optimized networks, to novel subjects suggests our modelscontain appropriate “inductive biases" that improve sample efficiency.
Revised: This remarkable generalization of response-optimized networks, butnot task-optimized networks, to novel subjects suggests our models contain appropriate “inductivebiases" that improve sample efficiency.

Original: Computational models generalize to OOD datasets We next investigated out-of-distribution(OOD) test settings to understand the generalization behavior of response-optimized models.
Revised: Computational models generalize to OOD datasets We next investigated out-of-distribution (OOD) testsettings to understand the generalization behavior ofresponse-optimized models.

Original: Thefirst OOD setting we consider is the NSD-synthetic experiment, comprising fMRI recordings tosynthetic stimuli (samples shown in Figure 3) from the same NSD subjects.
Revised: The first OOD setting weconsider is the NSD-synthetic experiment, comprisingfMRI recordings to synthetic stimuli (samples shown in Figure 3) from the same NSD subjects.

Original: As shown in Figure 3,response-optimized models significantly outperform task-optimized models(except inregion VO)in this OOD domain (‘artificial’ stimuli), suggesting that the mapping from task-optimized DNNfeatures to voxel responses, was domain/dataset-specific.
Revised: As shown in Figure 3, response-optimized models significantly outperformtask-optimized models in this OOD domain (‘artificial’stimuli), suggesting that the mapping from task-optimized DNN features to voxel responses, was domain/datasetspecific.

Original: Response-optimized models, on the otherhand, allow for flexible OOD generalization.
Revised: Response-optimized models, on the other hand,allow for flexible OOD generalization.

Original: Generalization to other novel fMRI datasets, including[49, 50, 51] is shown and discussed in the Appendix B.2.
Revised: Generalization toother novel fMRI datasets, including [27, 28, 29] is shownand discussed in the Suppl. Results.

Original: Learned spatial masks reproduce the fine-grained retinotopic organization of early visual cortex
Revised: Learned spatial masks reproduce the fine-grainedretinotopic organization of early visual cortex

Original: The factorized readout in the proposed model allows us to separate spatial and feature selectivity. We characterized the spatial tuning of early visual areas by estimating the population receptive field(pRF) from the learned spatial masks of the proposed response-optimized model; we compare theestimated parameters against measurements from the independentNSD pRF localizer experimentthat uses artificial stimuli. The pRF is defined as the region of the visual field within which a stimulusresults in increased aggregated activity across a population of neurons, as reflected in fMRI.
Revised: Next,we estimate the population receptive field from the learnedspatial masks and compare the estimated parametersagainst the measurements from the independent pRF localizer scan session.

Original: Basedon the analysis of these data, each brain voxel’s retinotopic organization can be described with threeparameters: the pRF location in polar coordinates (polar angle and eccentricity), and size.

Original: Usingthe voxel-wise spatial mask of the readout in our encoding model, we estimated the pRF location asthe average of the coordinates weighted by the learned spatial mask value at each grid location.

Original: Tocompare the estimated eccentricity against the corresponding measurements from pRF localizers,henceforth called ‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient betweenthe measured and predicted eccentricity arrays in each ROI.

Original: To compare the estimated polar anglevalues against corresponding measurements from pRF localizer scans, henceforth called ‘polar angleagreement’ , we adopted a measure from circular statistics called circular correlation coefficient [52],due to the circular nature of angular data (Appendix A.4).

Original: It is important to note that the encodingmodels do not account for the spatial locations of target voxels.
Revised: It is important to note that the encodingmodels do not account for the spatial locations of target voxels.

Original: Despite this, the estimated spatialtopographic organization from the encoding models exhibits remarkable levels of agreement withthe fine-scaled retinotopic organization of the cortex without any explicit supervision to do so(Figure 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angleagreement ∈[0.70-0.82].
Revised: Despite this, the estimated spatial topographic organization from the encoding models exhibits remarkable levels of agreementwith the fine-scaled retinotopic organization of the cortex without any explicit supervision to doso (Figure 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angleagreement ∈

Original: We further compared this agreement on the estimated pRF parametersfor the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linearreadouts from the fixed convolutional core of the response-optimized model (pre-trained on theinitial set of 4 subjects).
Revised: We further compared this agreement on the estimated pRF parametersfor the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linearreadouts from the fixed convolutional core of the response-optimized model (pre-trained on theinitial set of 4 subjects).

Original: We find that while the agreement increases with the sample size used tofit readouts, even with much fewer samples (e.g. 100), we can still characterize the retinotopicorganization (Figure 4C).
Revised: We find that while the agreement increases with the sample size used tofit readouts, even with much fewer samples (e.g. 100), we can still characterize their retinotopicorganization (Figure 4C).

Original: This is better than the retinotopic agreement observed for the CORnetS model (which also employed a factorized readout, enabling characterization of spatial tuning(Appendix Figure A.8)).

Original: Previously, retinotopic organization of the human visual cortex has beendominantly studied with spatially modulated stimuli.
Revised: Previously, retinotopic organization of the human visual cortex has beendominantly studied with spatially modulated stimuli.

Original: While encoding models have been shown tobe broadly useful for characterizing retinotopy [53, 31], a precise quantification of the agreementbetween model-estimated and localizer-estimated retinotopic parameters, as well as the effect ofsample sizes on this agreement, has been lacking.
Revised: While encoding models have been shown tobe broadly useful for characterizing retinotopy [30, 31], a precise quantification of the agreementbetween model-estimated and localizer-estimated retinotopic parameters, as well as the effect ofsample sizes on this agreement, has been lacking.

Original: Here, we present an alternate approach basedon naturalistic stimulation and response-optimized encoding models to delineate subject-specificretinotopic maps and precisely quantify the agreement.
Revised: Here, we present an alternate approach basedon naturalistic stimulation and response-optimized encoding models to delineate subject-specificretinotopic maps and precisely quantify the agreement.

Original: Remarkably, the retintopic organizationestimated from the proposed encoding models is even more spatially smooth than the localizer-basedorganization, despite not being informed by anatomy.
Revised: Remarkably, the retintopic organizationestimated from the proposed encoding models is even more spatially smooth than the localizer-basedorganization, despite not being informed by anatomy.

Original: visual cortical hierarchy
Revised: Response-optimized models characterize transformations of image representations along thevisual cortical hierarchy

Original: (a) Monotonically increasing alignment with human perception along the ventral visual stream
Revised: (a) Monotonically increasing alignment with human perception along the ventral visual stream

Original: We evaluated models on behavioral data from a Multiple Arrangement task where participants arranged NSD images according to the similarity of their content.
Revised: We evaluated models on behavioral data from a Multiple Arrangement task where participantsarranged NSD images according to the similarity of their content.

Original: This behavioral analysis of neuralnetwork models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capturehuman perceptual patterns and image semantics than early visual areas and this quantitative agreementincreases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’analysis discussed in Methods).
Revised: This behavioral analysis of neuralnetwork models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capturehuman perceptual patterns and image semantics than early visual areas and this quantitative agreementincreases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’analysis discussed in Methods).

Original: Importantly, responses patterns in their respective models groupsemantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region
Revised: Importantly, responses patterns in their respective models groupsemantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region

Original: VO).
Revised: VO).

Original: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes grouptogether in the left (Figure 5A), despite wide variations in the visual appearance of these categories.
Revised: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes grouptogether in the left (Figure 5A), despite wide variations in the visual appearance of these categories.

Original: This is remarkable, given that the models received no supervision from perceptual/semantic measures.
Revised: This is remarkable, given that the models received no supervision from perceptual/semantic measures.

Original: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Appendix Figure
Revised: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Suppl.

Original: A.10.
Revised: Results.

Original: This suggests that these models, solely optimized for high-level visual cortical responseprediction – without any semantic supervision – can even serve as powerful models for uncoveringthe structure of mental representations.
Revised: Thissuggests that these models, solely optimized for high-level visual cortical response prediction –without any semantic supervision – can even serve as powerful models for uncovering the structure of mental representations.

Original: Intriguingly, the representational structure of predictedresponses exhibits stronger agreement with perceived similarity than the representational structureof measured responses.
Revised: Intriguingly, the representational structure of predicted responses exhibitsstronger agreement with perceived similarity than the representational structure of measured responses.

Original: This suggests that both the perceptual and semantic information are likelymore explicit in predicted responses (even for held-out data) than measured responses.One possibleexplanation is
Revised: This suggests that both the perceptual and semantic information are likely more explicit in predictedresponses (even for held-out data) than measured responses.

Original: that

Original: fMRI measures of brain activity have undesired noise components and predictivemodels, in particular, response- optimized models as shown here, can denoise this 
Revised: One possible explanation is that fMRImeasures of brain activity have undesired noise components and predictive models, in particular,response-optimized models as shown here, can denoise this fMRI activity.

Original: fMRI

Original: activity (seealso Appendix Figure A.3).

Original: (b) Monotonically increasing separability of category information along the ventral visual

Original: stream Task-optimized computational models

Original: have

Original:  suggested that the ventral stream hierarchy ‘untangles’ representations through a sequence of processing steps, so that representations of objects and categories that are inseparable at the initial processing stage (e.g., V
Revised: Monotonically increasing separability of category information along the ventral visualstream Task-optimized computational models have suggested that the ventral stream hierarchy‘untangles’ representations through a sequence of processing steps, so that representations of objectsand categories that are inseparable at the initial processing stage (e.g., V1) become untangled at thelast stage of the hierarchy [32, 10].

Original: become untangled at the last stage of the hierarchy Here, we present a strong evidence for this increasedseparability through hypothesis-agnostic response-optimized models.
Revised: Here, we present a strong evidence for this increased separabilitythrough hypothesis-agnostic response-optimized models.

Original: Previous studies have shownthat the representational geometry for the same set of stimuli varies systematically across differentcortical areas, providing a useful signature for how information is transformed across differentcortical processing streams [54].
Revised: Previous studies have shown that therepresentational geometry for the same set of stimuli varies systematically across different corticalareas, providing a useful signature for how information is transformed across different corticalprocessing streams [32].

Original: Here, we employ the representational similarity analysis (RSA)framework to relate emergent representations in response-optimized models of difference brainregions against a ground truth adjacency matrix defined by object category labels.
Revised: Here, we employ the representational similarity analysis (RSA) framework to relate emergent representations in response-optimized models of difference brain regions againsta ground truth adjacency matrix defined by object category labels.

Original: Keeping modelarchitecture the same, simply by changing the response targets for an encoding model from voxelsin visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extracted representation.
Revised: Keeping model architecture thesame, simply by changing the response targets for an encoding model from voxels in visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extracted representation.

Original: The increasingly prominent block-diagonal structure in the RSMs along the ventralvisual stream (Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlightsthat the distributed activity patterns to exemplars of the same category become increasingly moresimilar; and patterns to exemplars of different categories become progressively dissimilar alongthe processing stream, strongly supporting the computational goal of the ventral visual stream asa categorization system.
Revised: The increasingly prominent block-diagonal structure in the RSMs along the ventral visual stream(Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlights that thedistributed activity patterns to exemplars of the same category become increasingly more similar; anddistributed patterns to exemplars of different categories become progressively dissimilar along theprocessing stream, strongly supporting the computational goal of the ventral visual stream as a visualcategorization system.

Original: We also quantified the separability of categorical information in differentvisual areas (Figure 5B).
Revised: We also quantified the separability of categorical information in differentvisual areas (Figure 5B).

Original: Here again, we see that this finding regarding increasing separability ofcategorical information holds across both datasets containing different visual categories.
Revised: Here again, we see that this finding regarding increasing separability ofcategorical information holds across both datasets containing different visual categories.

Original: Response-optimized models reveal high-level visual information within ventral-occipital cortex
Revised: Response-optimized models reveal high-level visual information within ventral-occipital cortex

Original: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model andwhether they are capable of discriminating between complex categories.
Revised: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model andwhether they are capable of discriminating between complex categories.

Original: Instead of looking at allmodel neurons emulating voxels, we built a compact neural representational space by passing a largestimulus set ( ∼ 27,000 images from the THINGS database [44]) through the predictive models andthen performing Principal Components Analysis (PCA) on the predicted responses of this network(Appendix A.6).
Revised: Instead of looking at allmodel neurons emulating voxels, we built a compact neural representational space by passing a largestimulus set ( ∼ 27,000 images from the THINGS database [20]) through the predictive models andthen performing Principal Components Analysis (PCA) on the predicted responses of this network(see Suppl. Methods).

Original: Visualizing the images from top concept categories that elicit highest and lowestactivation of individual principal components (PCs) reveals complex stimulus dimensions: the first PC, which explained ∼ 49% of the variance in responses strongly separates faces, animals andbodies from inanimate objects, despite vast variations in the visual appearance of images withinsub-categories.
Revised: Visualizing the images from top concept categories that elicit highest andlowest activation of individual principal components (PCs) reveals complex stimulus dimensions: thefirst principal component, which explained ∼ 49% of the variance in responses strongly separatesfaces, animals and bodies from inanimate objects, despite vast variations in the visual appearanceof images within sub-categories.

Original: The second PC, accounting for ∼ 16% variance, distinguished objects based on theirshape (circular vs. rectilinear), suggesting that such shape features are important for VO.
Revised: The second principal component, accounting for ∼ 16% variance,distinguished objects based on their shape (circular versus rectilinear objects), suggesting that suchshape features are important for VO.

Original: Importantly,these features do not emerge when the same analysis is applied to early visual ROIs (V1-V4).
Revised: Importantly, these features do not emerge when the sameanalysis is applied to early visual ROIs (V1-V4).

Original: Response-optimized models exhibita shape bias A striking known discrepancy between human and machine vision is the shape vs. texture bias: human classification decisions are shape-biased whereas CNNstrained for CV tasks tend to classifyimages by texture [55, 56].
Revised: Response-optimized models exhibit a shape bias A striking known discrepancy between humanand machine vision is the shape vs. texture bias: human classification decisions are shape-biasedwhereas CNNs trained for CV tasks tend to classify images by texture [24, 33].

Original: We studywhether the shape bias of humans isreflected in response-optimized models of cortical regions that subservevisual categorization in humans.
Revised: We study whetherthe shape bias of humans is reflected in response-optimized models of cortical regions that subservevisual categorization in humans.

Original: Wefirst train a linear layer on top ofthe predicted responses of responseoptimized models to map these responses to visual categories in the

Original: ImageNet-16 database [55] using asmall subset of this database. We thenassess the transfer performance ofresponse-optimized models for classification on held-out stimuli from the ImageNet-16 dataset as well as the Silhouette dataset described in [55](Appendix A.10).
Revised: First, we assess the transfer performance of response-optimizedmodels for classification on held-out stimuli from the ImageNet-16 dataset as well as the Silhouettedataset described in [24] (see Suppl. Methods).

Original: We find that theclassification performance is significantly better than what would beexpected from the same network architecture with a random initialization (Figure 6B), suggesting that thebrain response-optimization objectiveyields effective representations.
Revised: We find that the classification performanceis significantly better than what would be expected from the same network architecture with arandom initialization (Figure 6B), suggesting that a brain response-optimization objective alonecan yield effective representations.

Original: Further, this classification performance isbetter using models of intermediate and high-level visual areas (V4, LO and VO) than early visual areas, even approaching the transfer performance of an MS-COCO-trained object classification networkwith the same architecture (Figure 6B,top, see Appendix A.10 for model details), providing furthersupport for the separability of categorical information in high-level, but not early visual cortex [11].
Revised: Further, this classification performance is better using modelsof intermediate and high-level visual areas (V4, LO and VO) than early visual areas, providingfurther support for the separability of categorical information in high-level, but not early visualcortex [7].

Original: The LO model has the highest classification performance on the Silhouette dataset (which solelycontain shape information), consistent with and providing further evidence for previous hypothesesthat LO tracks object shape information [57].
Revised: The LO model has the highest classification performance on the Silhouette dataset (whichsolely contain shape information), consistent with and providing further evidence for the previoushypotheses that LO tracks object shape information [34].

Original: We then evaluate theclassification performance of thismapped response-optimized model on the diagnostictexture-shape cue conflict dataset[55] to assesswhether the model makes its categorization decisions based on shape or texture (Appendix A.10).
Revised: We then evaluate the performance of thismapped response-optimized model on the diagnostic cue conflict dataset to assess whether the modelmakes its categorization decisions based on shape or texture.

Original: Wefollow [55] and define shape bias as the percentage of times the model classifies an image based onits shape over texture, provided it returned either the correct shape or correct texture label for theimage.
Revised: We follow [24] and define shape biasas the percentage of times the model classifies an image based on its shape over texture, provided itreturned either the correct shape or correct texture label for the image.

Original: A model is shape-biased if its shape bias > 50 % .
Revised: A model is shape-biased if itsshape bias > 50 % .

Original: We observed that all response-optimizedmodels exhibit a shape bias (Figure 6B [bottom], Appendix Figure A.1), with this metric rangingfrom 56.7-69.4 % , suggesting that representations within the ventral visual stream are likely biasedtowards shapes than textures.
Revised: We observed that all response-optimized models exhibit a shape bias (Figure 6B[bottom]), with this metric ranging from 56.7-69.4 % , suggesting that representations within theventral visual stream are likely biased towards shapes than textures.

Original: This is in stark contrast to standard CV models which show atexturebias, for e.g., the same architecture trained for multi-label object classification on MS-COCO shows astrong texture bias and performs significantly worse on the Silhouettes dataset (Figure 6B, gray bar),suggesting that the distribution of NSD stimuli (which are drawn from MS-COCO) is not sufficientto drive the shape bias and the training objective (matching to neural responses rather than objectcategory labels) matters.
Revised: This is in stark contrast tostandard CV models which show a texture bias.

Original: Further, as shown in Figure 6B (bottom), it is not thelinear mapping toneural response but the way features are optimized that explains the differing shape/texture bias oftask-optimized and response-optimizedencoding models.
Revised: Further, as shown in Figure 6B (bottom), it isnot the mapping to neural response but the way features are optimized that explains the differingshape/texture bias of task-optimized and response-optimized models.

Original: We next performed a feature visualizationanalysis to find visual patterns that maximally activate individual model neurons emulating brainvoxels in  response-optimized encoding models.
Revised: We also visualized the features in different response-optimized models using an image synthesisprocedure (described in Methods).

Original: Following [58], we start with a random noise inputx 0 for each model neuron i and iteratively update the input along the gradient ∂a i ∂x to synthesizeinputs that would result in higher and higher predicted activation a i for that neuron (Appendix A.9).

Original: Visual inspection shows that the features increase in complexity as we go from early to high-levelvisual ROI models: simple edge-like features or high-low frequency detectors are seen in V1, curvesand edges are seen in V2, more complex curvatures in V4 and finally, fully-formed complex shapeslike concentric hexagons are seen in LO and VO (Fig. 6A, Appendix Fig. A.7). We also validated thisincreased complexity using quantitative measures based on compression ratios (Appendix Table A.3).
Revised: Visual inspection shows that the features increase in complexityas we go from early to high-level visual ROI models: simple edge-like features or high-low frequencydetectors are seen in V1, curves and edges are seen in V2, more complex curvatures in V4 and finally,fully-formed complex shapes like concentric hexagons are seen in LO and VO (Figure 6A).

Original:  Discussion
Revised:  Discussion

Original: In this study, we leveraged the large sample size and natural variation represented in the Natural Scenes Dataset to train deep neural networks directly on  the brain response.
Revised: Here, we set out to answer the question: How far can deep learning on neural responses alone get usin predicting neural responses to ID and OOD test sets and understanding the ventral visual pathway?

Original: Concurrently, recentpapers [31, 30] have also demonstrated that when trained with large amounts of data such as NSD,the response-optimization approach can perform competitively with state-of-the-art task-optimizedmodels (discussed in more detail in Appendix C.1).
Revised: Thus, far, visual neuroscience had been largely measurement-limited, constraining modeling effortsthat bring neuroscience data to bear on the model development process. The large sample size of Natural Scenes Dataset presented an opportunity to change this status quo. We capitalized on thenatural variation in NSD to train deep neural networks solely with supervision from brain responseprediction.

Original: We studied model generalization in varioussettings: from synthetic stimuli to novel fMRI datasets with  different preprocessing.
Revised: We studied model generalization in different settings: from synthetic stimuli to novel fMRIdatasets with widely different preprocessing pipelines.

Original: Our resultsrecapitulated key features of early visual areas (e.g. their precise retinotopic organization) andrepresentational properties in higher level areas.
Revised: We demonstrated success in recapitulatingkey features of early visual areas (e.g. their precise retinotopic organization), representationaltransformations along the ventral visual stream along with demonstrating that high-level visualfeatures like faces, places and complex shapes are represented in high-level visual areas.

Original: We also demonstrated that response-optimizedmodels can capture human behavior, including not only object categorization, but also other finegrained measures such as perceived image similarity.
Revised: Remarkably,we also demonstrated that despite no supervision from human behavioral measures, these modelscan capture human behavioral abilities, including not only object categorization, but also otherfine-grained measures such as perceived image similarity.

Original: The response-optimization approach, inprinciple, allows exploring infinite paths in the model space, and when optimized properly, convergingtowards one that explains maximum possible variance in the data.
Revised: These hypothesis-agnostic models, webelieve, can complement task-optimized models in furthering our understanding of human vision; theresponse-optimization approach, in principle, allows exploring infinite paths in the model space, andwhen optimized properly, converging towards one that explains maximum possible variance in thedata.

Original: Since any features that emergein these optimized networks are a result of optimization to match neural targets, unconfounded byany other supervision signal, these models can provide stronger tests about the visual representationswithin different visual areas.
Revised: Since any features that emerge in these optimized networks are a result of optimization to matchneural targets, unconfounded by any other supervision signal, these models can provide stronger testsabout the visual representations within different visual areas.

Original: In terms of human object categorization behavior, our models lack significantly compared to ImageNet-supervised DNN models.
Revised: In terms of human object categorization behavior (as tested on ImageNet-16), our models lacksignificantly compared to ImageNet-supervised DNN models.

Original: While part of this may be due to the domain shift, the restricted transfer learning setting (a single linear layer mapping), small size of the transfer set orsimply the coarse spatial resolution of fMRI, this gap is  worth exploring.Further, using neural data tobuild better models for other communities (e.g. computer vision) is another scientifically fascinatingdirection and can develop reinforcing cycles.
Revised: While part of this may be due to thedomain shift (NSD/COCO → ImageNet), the restricted transfer learning setting (a single linear layerto map predicted voxel responses to category labels), small size of the transfer set or simply the coarsespatial resolution of fMRI, this gap is still worth exploring.

Original: Studies have shown that explicitly encouraging neuralnetworks to build representations similar V1 representations can improve their robustness undernoise and adversarial attacks [59, 60].

Original: In a similar vein, regularization with high-level visual corticalactivity might help grant models brain-like biases (e.g. shape bias) and close the gap between humanand machine vision.

Original: Alternatively, probing the connections in response-optimized models, whichalready exhibit a shape bias, using model pruning and attribution methods can yield a mechanisticunderstanding of the shape bias in neural activity and reveal principles that may enable CV modelsto attain similar representations.

Original: Recent studies have inspired optimism that studying connectionweights to reveal underlying computational principles may be tractable [33, 61].

Original: Checklist

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope? [Yes] (b) Did you describe the limitations ofyour work? [Yes] They are described within the
Revised: In this study, we focused on the use of

Original: Discussion section

Original: (c) Did you discuss any potential negative societal impacts of your work? [N/A] We don’t foresee any negative societal impacts of this work. (d) Have you read the ethics review guidelines and ensured that your paper conforms to 
Revised: DNN models in understanding the ventral visual stream but using neural data to build better modelsfor other communities (e.g. computer vision) is also scientifically fascinating and has the potential todevelop reinforcing cycles.

Original: them?

Original: [Yes] I’ve read all guidelines

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical
Revised: It will be interesting to see whether regularizing object categorizationmodels with model-predicted neural responses can grant them brain-like biases (e.g. shape bias)and capabilities (e.g. OOD generalization).

Original: results are discussed in this paper.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: No theoretical results

Original: are discussed in this paper.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Yes, the code is
Revised: Studies have shown that explicitly encouraging neuralnetworks to build representations that are like neural representations or principles in V1 can improvetheir robustness under stimulus noise and adversarial attacks [35, 36].

Original: included in the Supplementary Material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were

Original: chosen)?

Original: [Yes] Yes, all training details are included in the Methdos and Supplementary

Original: Methods.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: [Table]

Original: ments multiple times)?

Original: [Yes] Yes, they are displayed in all figures.

Original: [Table]

Original: (d) Did you include the total amount of compute and  the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] Yes. All models were trained,validated and analysed on an NVIDIA Titan Xp GPU. Training each response-optimizedmodel on 1 GPU took ∼ 8 hours.
Revised: In a similar vein, regularizationwith high-level visual cortical activity might help close the gap between human and machine vision,since this activity is the neural basis for human visual intelligence.

Original: Additional details are listed in the Supplementary.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] e.g. [47, 62]

Original: [Table]

Original: (b) Did you mention the license of the assets?

Original: [No] The assets employed in this study were

Original: [Table]

Original: made publicly available by the creators

Original: [Table] (c) Did you include any new assets either in the supplemental material or as a URL? [No] [Table]
Revised: A Appendix

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [No] The assets employed in this study were made publicly availableby the creators

Original: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: [No] The assets don’t contain any offensive contentto our knowledge.

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)

Original: Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?

Original: [N/A](c)

Original: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]