Original: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models
Revised: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Decades of experimental research based on simple, abstract stimuli has revealedthe coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complexvisual categories in the anterior ventral stream.
Revised: Decades of experimental research based on simple, abstract stimuli has revealedthe coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complexvisual categories in the anterior ventral stream.

Original: However, these studies are, byconstruction, constrained by their a priori hypotheses.
Revised: However, these studies are, byconstruction, constrained by their a priori hypotheses.

Original: Furthermore, beyond theearly stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.
Revised: Furthermore, beyond theearly stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.

Original: In this work, wepropose to employ response-optimized encoding models trained solely to predictthe functional MRI activation, in order to gain insights into the tuning propertiesand representational transformations in the series of areas along the ventral visualpathway.
Revised: In this work, wepropose to employ response-optimized encoding models trained solely to predictthe functional MRI activation, in order to gain insights into the tuning propertiesand representational transformations in the series of areas along the ventral visualpathway.

Original: We demonstrate the strong generalization abilities of these models onartificial stimuli and novel datasets.
Revised: We demonstrate the strong generalization abilities of these models onartificial stimuli and novel datasets.

Original: Intriguingly, we find that response-optimizedmodels trained towards the ventral-occipital and lateral-occipital areas, but notearly visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.
Revised: Intriguingly, we find that response-optimizedmodels trained towards the ventral-occipital and lateral-occipital areas, but notearly visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.

Original: We further probe the trainednetworks to reveal representational biases in different visual areas and generateexperimentally testable hypotheses.
Revised: We further probe the trainednetworks to reveal representational biases in different visual areas and generateexperimentally testable hypotheses.

Original: Our analyses suggest a shape-based processingalong the ventral visual stream and provide a unified picture of multiple neuralphenomena characterized over the last decades with controlled fMRI studies.

Original:  Introduction
Revised:  Introduction

Original: Understanding the nature of representations and computations in the visual system has been alongstanding goal in neuroscience.
Revised: Understanding the nature of representations and computations in the visual system has been alongstanding goal in neuroscience.

Original: Substantial progress been madein this quest for early stages ofthe visual cortex by presenting model organisms with simple, abstract stimuli like noise or sine-wavegratings and studying the evoked responses, such as the peak of their tuning curves.
Revised: Substantial progress been made by presenting model organismswith simple, abstract stimuli like noise or sine-wave gratings and studying the evoked responses.

Original: These carefullycrafted experiments, for instance, revealed the presence of edge detectors in the primary visualcortex [1] and sparked the search for the ‘optimal input’ beyond early visual areas across the visualcortical hierarchy.
Revised: These carefully crafted experiments, for instance, revealed the presence of edge detectors in theprimary visual cortex [1] and sparked the search for the ‘optimal input’ beyond early visual areasacross the visual cortical hierarchy.

Revised: Significant efforts have also been made to understand the featurecomplexity of tuning in V4 and IT neurons [2, 3, 4, 5].

Original: Over the last two decades, through numerous experiments, recording and analysistechniques, a conceptual understanding has emerged that early visual areas extract low-level featureslike edges or curves, mid-level regions extract complex local shapes, and high-level regions encodeand represent different semantic categories, like faces or scenes [2, 3, 4, 5, 6].
Revised: Over the last two decades, through numerousexperiments, recording and analysis techniques, a conceptual understanding has emerged that earlyvisual areas extract low-level features like edges or curves, mid-level regions extract complex localshapes, and high-level regions encode semantic categories, like faces or scenes [6, 7, 8, 9, 10].

Original: While the underlyingrepresentational transformations that enable flexible and efficient visual perception remain unknown,there is accumulating evidence that visual categorization occurs in the ventral temporal cortex inhumans [7].
Revised: Whilemuch of details remain unknown, there is accumulating evidence that visual categorization occursin the ventral temporal cortex in humans [11].

Original: This understandingof the visual system was enabled by synthesizing findings acrossseveral different hypothesis-drivenexperimental studies, most of them employing artificial stimuliatypical of real-word vision, thus lacking ethological validity.
Revised: This understanding was enabled by synthesizingfindings across different hypothesis-driven studies, most of them employing artificial stimuli atypicalof real-word vision, thus lacking ethological validity.

Original: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks, haveemerged as the most accurate models of the primate ventral visual stream [8].
Revised: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks,have emerged as the most accurate models of the primate ventral visual stream [12, 13, 14, 15].

Original: Furthermore, these CVtask-optimized models offer a hierarchical correspondence to visual areas along the ventral stream:early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [8, 9].
Revised: Furthermore, these CV task-optimized models offer a hierarchical correspondence to visual areasalong the ventral stream: early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [12, 16, 17, 18, 19, 20, 21, 22].

Original: They provide intelligibility in the form of the goal-driven perspective, highlighting the functionalrole of representations [10].
Revised: They provide intelligibility in the formof the goal-driven perspective, highlighting the functional role of representations [23].

Original: However, we are far from a complete understanding of the top-downcomputational objectives (the battery of ethologically-relevant tasks) that may have shaped neuralrepresentations through evolution or experience-based learning; in the absence of right hypotheses,the task-optimized approach can fall short.
Revised: However,we are far from a complete understanding of the top-down computational objectives (the batteryof ethologically-relevant tasks) that may have shaped neural representations through evolution orexperience-based learning; in the absence of right hypotheses, the task-optimized approach canfall short.

Revised: Studies have already revealed remarkable inconsistencies between these models andhuman behavior and gaps remain in their respective robustness to distortions, image-level consistencyetc [24].

Revised: Recently, the idea of using neural data in the model development process has gained sometraction, and demonstrated success in both increasing model-human behavior alignment [25, 26, 27],as well as increasing the model-brain alignment while revealing novel insights about informationprocessing in neural systems [28, 29, 30, 31].

Original: In this study, we leverage recent advances in large-scale fMRI data collection [11] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.
Revised: In this study, we leverage recent advances in large-scale fMRI data collection [32] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.

Original: Importantly,the stimulus set employed in this study contains crowded images of multiplecommon objects in theirnatural contexts, thus being more typical of everyday scenes and allowing us to characterize neuralrepresentations and computations in rich, naturalistic conditions.
Revised: Importantly,the stimulus set employed in this study contains crowded images of multiple objects in their natural contexts, thus being more typical of everyday scenes and allowing us to characterize neuralrepresentations and computations in rich, naturalistic conditions.

Original: We demonstrate the remarkableability of these “response-optimized” neural encoding models in reproducing the stimulus-responserelationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassingthe unprecedented prediction performance of task-optimized models.
Revised: We demonstrate the remarkableability of these “response-optimized” neural encoding models in reproducing the stimulus-responserelationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassingthe unprecedented prediction performance of task-optimized models.

Original: We show that these modelsgeneralize well to contexts that are vastly different from their training conditions (artificial stimuli,novel subjects, etc.), providing a strong quantitative validation for the response-optimization approachof building encoding models for neurobiological visual systems beyond the retina [12].
Revised: We show that these modelsgeneralize well to contexts that are vastly different from their training conditions (artificial stimuli,novel subjects, etc.), providing a strong quantitative validation for the response-optimization approachof building encoding models for neurobiological visual systems beyond the retina [33].

Original: Our approachin this paper, however, goes beyond prediction accuracy and aims to gain insights about neuronaltuning properties.
Revised: Our approachfurther goes beyond prediction accuracy and aims to gain insights about neuronal tuning properties.

Original: Specifically, we implement neural encoding models that disentangle the processingof the “what” and the “where.”
Revised: Specifically, we implement neural encoding models that disentangle the processing of the “what” andthe “where.”

Original: This allows us to reveal the selectivity of individual brain voxels alongboth location and appearance dimensions.
Revised: This allows us to reveal the selectivity of individual brain voxels along both locationand appearance dimensions.

Original: We first demonstrate that the estimated spatial topographicorganization from our neural encoding models exhibits remarkable agreement with the fine-scaledretinotopic organization of the cortex, which is typically measured with dedicated, painstakingexperimental paradigms.
Revised: We first demonstrate that the estimated spatial topographic organizationfrom our neural encoding models exhibits remarkable agreement with the fine -scaled retinotopicorganization of the cortex, which is typically measured with dedicated, painstaking experimentalparadigms.

Original: We analyze the features that emerge in the neural encoding models tounderstand the tuning properties of cortical voxels.
Revised: We analyze the features that emerge in the neural encoding models to understand the tuning properties of cortical voxels.

Original: This analysis suggests shape-biased tuning acrossthe ventral visual stream, increasing in complexity from Gabor-like features in V1, curves and anglesin V2 to concentric (circular) and complex shape processing in V4 and higher-order lateral-occipitaland ventral-occipital ROIs respectively.
Revised: This analysis suggests shape-biased tuning across the ventral visualstream, increasing in complexity from Gabor-like features in V1, curves/angles in V2 to concentricand complex shape processing in V and higher-order lateral -occipital and ventral-occipital ROIsrespectively.

Original: We also go beyond single-voxel analysis to characterizehow representations evolve along the ventral visual pathway.
Revised: We also go beyond single -voxel analysis to characterize how representations evolvealong the ventral visual pathway.

Original: Probing the information encoded inmodel-predicted patterns of activity reveals the monotonically increasing separability of categoryinformation along the ventral stream hierarchy, strongly supporting the computational goal of theventral visual stream as a visual categorization system.
Revised: Probing the information encoded in model-predicted patterns ofactivity reveals the monotonically increasing separability of category information along the ventralstream hierarchy, strongly supporting the computational goal of the ventral visual stream as a visualcategorization system.

Original: We further demonstrate the correspondencebetween model representations (of high-level visual areas) and mental representations (as assessedwith human similarity judgements), suggesting that these response-optimized models inherentlycapture human psychological judgements, without any explicit supervision to do so.
Revised: We further demonstrate the correspondence between model representations (ofhigh-level visual areas) and mental representations (as assessed with human similarity judgements),suggesting that these response-optimized models inherently capture human psychological judgements,without any explicit supervision to do so.

Original: Furthermore, aprinciple components analysis reveals that models of high-level visual ROIs capture complex stimulusdimensions like faces, bodies and shapes.
Revised: Furthermore, a principle components analysis reveals thatmodels of high-level visual ROIs capture complex stimulus dimensions like faces, bodies and shapes.

Original: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, yet naturalisticdatasets can be used to confirm neural phenomena characterized painstakingly using controlledexperimental paradigms that might be hard to replicate in individual subjects.
Revised: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, naturalistic datasetscan be used to confirm neural phenomena characterized painstakingly using controlled experimentalparadigms that might be hard to replicate in individual subjects.

Original: More importantly,however, we show how in silico experiments on neural encoding models can generate novel hypothesesand provide a powerful framework for building broad, generalizable theories about neural informationprocessing.
Revised: More importantly, however, we showhow in silico experiments on neural encoding models can generate novel hypotheses and provide apowerful framework for building broad, generalizable theories about neural information processing.

Original:  Materials and Methods
Revised:  Materials and Methods

Original: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is providedelsewhere [11] (see also Supplemental Methods).
Revised: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is providedelsewhere [32] (see also the Appendix).

Original: TheNSD dataset contains measurements of fMRIresponses from 8 participants who each viewed 9,000–10,000 natural scenes (22,000–30,000 trials)over the course of 30–40 sessions.
Revised: The dataset contains measurements of fMRI responses from8 participants who each viewed 9,000–10,000 natural scenes.

Original: A special set of 1,000 images were shared across subjects; theremaining were mutually exclusive across subjects.
Revised: A special set of 1,000 images wereshared across subjects; the remaining were mutually exclusive.

Original: We focused on modeling responses within 5 visualcortical ROIs in the study.
Revised: We focused on modeling responseswithin 5 visual cortical ROIs in the study.

Original: Three ROIs belonging to the retinoptic early visual cortex, namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session for each subject [11].
Revised: Three ROIs belonging to the retinoptic early visual cortex,namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session [32].

Original: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipitalareas (LO1-2) were delineated using a visual probabilistic atlas [13](Figure 1B)
Revised: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipitalareas (LO1-2) were delineated using a visual probabilistic atlas [34](Figure 1B).

Original: Response-optimized neural encoding model
Revised: Response-optimized neural encoding model

Original: We trained separate voxel-level neural encodingmodels for each of the above regions with the same backbone architecture.
Revised: We trained separate voxel-level neural encodingmodels for each of the above regions with the same backbone architecture.

Original: Keeping the architectureconstant across visual areas ensured that differences in emergent representations are driven mainlyby the nature of response patterns.
Revised: Keeping the architectureconstant across visual areas ensured that differences in emergent representations are driven mainlyby the nature of response patterns.

Original: The predictive model comprises a convolutional neural network(CNN) feature extractor shared across all subjects but unique to specific visual areas.
Revised: The predictive model comprises a convolutional neural network(CNN) feature extractor shared across all subjects, but unique to specific visual areas.

Original: We employ alinear readout model on top of the feature space to predict the responses of individual voxels in aspecific region.
Revised: We employ alinear readout model on top of the feature space to predict the responses of individual voxels in aspecific region.

Original: The linear readout is factorized into spatial and feature dimensions following [14](Figure 1A).
Revised: The linear readout is factorized into spatial and feature dimensions following [35](Figure 1A).

Original: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).
Revised: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).

Original: The shared CNN feature extractor consists of foursequential convolutional blocks, with each blockcomprising the following feedforward computations: two 3 × 3 convolutional layers, each followedby an inner batch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.
Revised: The shared CNN feature extractor consists of four convolutional blocks, with each block comprisingthe following feedforward computations: two 3 × 3 convolutional layers, each followed by an innerbatch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.

Original: Instead of regular convolutions, we employ E(2)-steerable convolutions in all our models to extractfeatures invariant to orientation [15].
Revised: We employ E(2)-steerable convolutions in all our models to extract features invariant to orientation [36].

Original: This modeling choice is inspired by neural computations inearly visual areas where groups of neurons are known to perform similar computations, e.g., edge orcurve detection, at different orientations.
Revised: Thismodeling choice is inspired by neural computations in early visual areas where groups of neurons areknown to perform similar computations, e.g., edge or curve detection, at different orientations.

Original: Each convolutional layer contains 48 feature sets extractedat 8 orientations.
Revised: Eachconvolutional layer contains 48 feature sets extracted at 8 orientations.

Original: Weights of the readout are a computed as outer products between a spatial filter anda feature vector.
Revised: Weights of the readout are acomputed as outer products between a spatial filter and a feature vector.

Original: The spatial filter further has a positivity constraint (enforced using rectification) andis length-normalized for each voxel.
Revised: The spatial filter further hasa positivity constraint (enforced using rectification) and is length-normalized for each voxel.

Original: Training and testing response-optimized encoding models We used 4 NSD subjects for trainingand reserved the other 4 subjects for studying generalization.
Revised: Training and testing response-optimized encoding models We used 4 NSD subjects for trainingand reserved the other 4 subjects for studying generalization.

Original: The 4 training subjects collectivelysaw 37,000 natural scene images, including the 1,000 shared images.
Revised: The first group collectively saw 37,000natural scene images, including the 1,000 shared images.

Original: We used the 1,000 sharedimages for testing our models and split the remaining stimulus set into 35,000 training and 2,000validation images.
Revised: We used the 1,000 shared images for testingour models and split the remaining stimulus set into 35,000 training and 2,000 validation images.

Original: All parameters of the neural encoding model were optimized jointly to minimizethe masked mean squared error between the predicted and measured fMRI response.
Revised: All parameters of the neural encoding model were optimized jointly to minimize the masked meansquared error between the predicted and measured fMRI response.

Original: This loss allowsus to propagate errors through the shared CNN even if the subjects are not exposed to commonstimuli since we can exclude the subjects/voxels for which the target response is not present from theloss calculation.
Revised: This loss allows us to propagateerrors through the shared CNN even if the subjects are not exposed to common stimuli since we canexclude the subjects/voxels for which the target response is not present from the loss calculation.

Original: We quantify ‘predictive accuracy’ on the test images as the Pearson’s correlationcoefficient between the predicted and measured fMRI response at each voxel.
Revised: Wequantify ‘predictive accuracy’ on the test images as the Pearson’s correlation coefficient between thepredicted and measured fMRI response at each voxel.

Original: We also compare thisagainst the noise-ceiling (see Suppl.
Revised: We also compare this against the noise-ceiling,expressed as the signal-to-noise ratio when considering the response variability of each voxel across3 repetitions per image (see Appendix A.5).

Original: Methods for further details).
Revised: Table A.1 provides a summary of all analyses methodsand baselines used for evaluating and interpreting the proposed response-optimized models. Baseline models (a) Category ideal observer model: We fit a non-image-computable modelusing human-generated annotations for NSD images (obtained from the MS-COCO database). Theinput to the categorical model is a k-hot encoding vector corresponding to the 80 object categoriesannotated in the database, where each element indicates the presence/absence of the correspondingcategory in the image.

Original: Generalization to new subjects We tested the generalization of response-optimized models  to 4test subjects in NSD by freezing the weights of the shared CNN and only training linear readouts. Wevaried the size of the training set for optimizing linear readout weights from a mere 100 stimulusresponse pairs to a large set of 4,500 pairs from novel subjects (see Suppl.
Revised: We fitted l 2 regularized linear regression models on this space to predictvoxel responses for held-out stimuli (Appendix A.8). (b) Task-optimized Models: We investigatedseveral DNN models, including AlexNet [37], ResNet-50 [38], DenseNet [39] and CORnet-s [40],optimized for object recognition on the large-scale ImageNet dataset [41] (Appendix A.3, Table

Original: Methods).
Revised: A.2).

Original: Characterizing spatial tuning of

Original: early visual areas The factorized readout allow us to separatespatial and feature selectivity. We assessed whether the spatial selectivity of our model trained onnatural stimuli agrees with measurements fromcarefully controlled experiments that use artificialstimuli. The pRF is defined as the  region  of the visual field within which a stimulus results inincreased aggregated activity across a population of neurons, as reflected in fMRI.
Revised: Since these models can demonstrate lower prediction performance due to the domain shiftbetween ImageNet and NSD stimuli, we also employed a ResNet-50 architecture pre-trained forobject detection on the MS-COCO dataset [42] as an additional baseline (since NSD stimuli weredrawn from the MS-COCO dataset). For CORnet-S, we pre-select the layer homologous to eachvisual region and employ a spatial × feature factorized readout (same as the response-optimizedmodel) on top of the fixed feature backbone, allowing a more direct comparison to the proposed model.

Original: It is commonlyestimated in a retinotopy experiment using pRF localizer scans. Based on the analysis ofthesedata, each brain voxel’s retinotopic organization can then be described with three parameters:
Revised: For the remaining models, we followed the more standard procedure for fitting task-optimizedmodels to voxel responses based on a full linear readout. We extracted features from pre-trainedlayers of each network and fitted l 2 regularized linear regression models separately for each layer.

Original: thepRF location in polar coordinates (polar angle and eccentricity), and size.
Revised: The regularization parameter was optimized independently for each subject, each layer, each modeland for voxels in each visual area by testing among 8 log-spaced values in [1e-4, 1e4].

Original: Using the voxel-wisespatial mask of the readout in our encoding model, we estimated the pRF location as the average ofthe coordinates weighted by the learned spatial mask value at each grid location.
Revised: For each visualarea and model, we identified the layer that best predicts responses on the validation set. These modellayers were used to quantify test performance.

Original: To compare theestimated eccentricity against the corresponding measurements from pRF localizers, henceforth called‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient between the measuredand predicted eccentricity arrays in each ROI. To compare the estimated polar angle values againstcorresponding measurements from pRF localizer scans, henceforth called ‘polar angle agreement’ ,we adopted a measure from circular statistics called circular correlation coefficient [18], due to thecircular nature of angular data (see Supp. Methods).
Revised: (c) Response-optimized rotation symmetries):

Original:  Characterizing neural representations by behavioral comparisons: (a) Assessing separabilityof category and semantic information along the visual hierarchy In addition to probing thelearned readout layer that gives insights into spatial tuning properties of voxels, we were interested inthe analysis of the response-optimized feature extractor for each visual area to understand featuraltuning. To achieve this, we used representational similarity analysis (RSA) [19].
Revised: Here, we train response-optimized models with the same architecture as the proposed models butwithout sharing weights across filter orientations. Characterizing neural representations by behavioral comparisons: (a) Assessing separabilityof category and semantic information along the visual hierarchy To probe the learned featuraltuning in the response-optimized model for each visual area, we used representational similarityanalysis (RSA) [43].

Original: Here, we correlatedthe patterns of predicted activity between multiple exemplars from different categories of stimuli toobtain a representational similarity matrix (RSM) for each visual area.
Revised: Here, we correlated the patterns of predicted activity between multipleexemplars from different categories of stimuli to obtain a representational similarity matrix (RSM)for each visual area.

Original: Further, we quantified theseparability of categorical information within these similarity matrices by computing a correlationcoefficient (Kendall’s τ ) between model RSMs against a ground truth adjacency matrix definedby object category labels.
Revised: Further, we quantified the separability of categorical information within thesesimilarity matrices by computing a correlation coefficient (Kendall’s τ ) between model RSMs againsta ground truth adjacency matrix defined by object category labels.

Original: Specifically, the elements of this matrix are 1 if the correspondingtwo images belong to different categories and 0 if the images belong to the same category.
Revised: Specifically, the elements of thismatrix are 1 if the corresponding two images belong to different categories and 0 if the imagesbelong to the same category.

Original: Thisanalysis was first performed using a subset of 104 images from the THINGS database [20] thatbelong to a pre-defined set of categories for ease of visualization.
Revised: This analysis was first performed using a subset of 104 images fromthe THINGS database [44] that belong to a pre-defined set of categories for ease of visualization.

Original: These categories were chosen inaccordance with previous fMRI studies employing RSA [21, 22] and include the following: { face,hand, elephant, cat, plant, fruit, car, tool } .
Revised: These categories were chosen in accordance with previous fMRI studies employing RSA [45, 46]and include the following: { face, hand, elephant, cat, plant, fruit, car, tool } .

Original: Subsequently, we also validated the obtained results usinganother large-scale vision dataset, namely ImageNet-16 [23].
Revised: Subsequently, we alsovalidated the obtained results using another large-scale vision dataset, namely ImageNet-16 [47].

Original: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images and computed the similarity betweenmodel RSMs and the categorical adjacency matrix following the same approach described for the THINGS dataset.
Revised: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images andcomputed the similarity between model RSMs and the categorical adjacency matrix following thesame approach described for the THINGS dataset.

Original: To assess visual information flow in relation to high-level semantic understanding,we computed pairwise similarity between predicted responses for held-out NSD images to construct model-predicted RSMs for each visual area.
Revised: To assess visual information flow in relation tohigh-level semantic understanding, we computed pairwise similarity between predicted responses forheld-out NSD images to construct model-predicted RSMs for each visual area.

Original: We constructed a semantic RSM by correlating the80-D binary category vectors (which indicate whether each category was present in the image or not)for every pair of these NSD images and quantified agreement with model-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).
Revised: We constructed asemantic RSM by correlating the 80-D binary category vectors (which indicate whether each categorywas present in the image or not) for every pair of these NSD images and quantified agreement withmodel-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).

Original: We computed this agreement usingmeasured RSMs for all visual areas as well.
Revised: Wecomputed this agreement using measured RSMs for all visual areas as well.

Original: In RSA described here and anywhere else, error bars(95 % CI) are computed by performing bootstrap sampling over voxels.
Revised: In RSA described hereand anywhere else, error bars (95 % CI) are computed by performing bootstrap sampling over voxels.

Original:  Assessing alignment with human perception We compared model representations againstbehavioral data from a Multiple Arrangement task where participants interpreted 100 NSD imagesand arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ wasreleased with NSD [11]).
Revised: (b) Assessing alignment with human perception We compared model representations againstbehavioral data from a Multiple Arrangement task where participants interpreted 100 NSD imagesand arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ wasreleased with NSD [32]).

Original: Critically, these images were not seen by any of the response-optimizedmodels.
Revised: Critically, these images were not seen by any of the response-optimized models.

Original: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROIusing the correlation distance metric for quantifying pairwise dissimilarities between model-predictedvoxel responses.
Revised: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROIusing the correlation distance metric for quantifying pairwise dissimilarities between model-predictedvoxel responses.

Original: We compared model RDMs against human pairwise dissimilarity judgements usingthe Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.
Revised: We compared model RDMs against human pairwise dissimilarity judgements usingthe Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.

Original: Characterizing shape bias of model representations We first train a linear layer on top of thepredicted responses of response-optimized models to map these responses to visual categories inthe ImageNet-16 database using a small subset of this database.

Original: Next, we evaluate the classificationperformance of these linear classifiers on held-out images from the ImageNet-16 dataset and Silhouettes dataset [24].

Original: Finally, we quantify the shape bias of all models using 1,200 images from thetexture-shape cue conflict dataset [24] (see Suppl. Methods).

Original: Characterizing voxel tuning through feature visualization We performed a feature visualizationanalysis to find visual patterns that would maximally activate individual model neurons emulatingbrain voxels in different response-optimized encoding models.

Original: Following [25], we start with arandom noise input x 0 for each model neuron i and iteratively update the input along the gradient∂a i ∂x to synthesize inputs that would result in higher and higher predicted activation a i for that neuron(see Suppl.

Original: Methods).

Original:  Results
Revised:  Results

Original: Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B).
Revised: Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B, Appendix Figure A.2, A.6).

Original: Next, we investigated the possible quantitative advantages of learning a rotationally-symmetric featurespace against the representations from a standard convolutional architecture with no weight sharingacross orientations.
Revised: Next, we investigated the possible quantitative advantages of learninga rotationally-symmetric feature space against the representations from a standard convolutionalarchitecture with no weight sharing across orientations.

Original: Importantly, we observe that the proposed rotationally-symmetric convolutionalarchitecture performs on par with a standard CNN (Figure 2A), even outperforming the latter incertain visual ROIs, despite containing fewer parameters.
Revised: Importantly, we observe that the proposedrotationally-symmetric convolutional architecture performs on par with a standard CNN (Figure 2A),even outperforming the latter in certain visual ROIs, despite containing fewer parameters.

Original: This suggests that sharing weights acrossfilter orientations provides a strong inductive bias, allowing us to fit expressive models efficiently.
Revised: Thissuggests that sharing weights across filter orientations provides a strong inductive bias, allowing usto fit expressive models efficiently.

Original: Wefurther compared these models against semantic and task-optimized models, and found that responseoptimized models consistently outperform these baselines in early visual areas.
Revised: We further compared these models against ideal observer andtask-optimized models, and found that response-optimized models consistently outperform thesebaselines in early visual areas.

Original: In high-level visualareas, the response-optimized model  outperforms the semantic model, while performing on par withtask-optimized model.
Revised: In high-level visual areas, the response-optimized model significantly outperforms the ideal observer model, while performing on par with or even surpassing sometask-optimized models.

Original: Next, we assessed how each of these predictive models generalizes to the remaining set of 4 testsubjects.
Revised: Next, we assessed how each of these predictivemodels generalizes to the remaining set of 4test subjects.

Original: This analysis intended to compare the statistical efficiency afforded by different representations [26].
Revised: This analysis intended to comparethe statistical efficiency afforded by differentrepresentations [48].

Original: As such, we examined the trend of how much stimulus-response data the models need toachieve a certain level of prediction performance [26].
Revised: As such, we examinedthe trend of how much stimulus-response datathe models need to achieve a certain level ofprediction performance [48].

Original: Here, we freeze the shared feature extractornetwork and vary the amount of stimulus-response pairs from the 4 new subjects to train the linearreadouts.
Revised: Here, we freeze the shared feature extractor network and vary the amount of stimulus-responsepairs from the 4 new subjects to train the linear readouts (Appendix A.7).

Original: We do the same for the task-optimized baseline, where the linear prediction model is fiton new subjects.
Revised: We do the samefor the task-optimized baselines, where the linear prediction model is fit on new subjects.

Original: As shown in Figure 2, the response-optimized models can achieve a significantlyhigh accuracy even when only just 5 % of the training examples from novel subjects ( ∼ 500 samples)are used to train the readout.
Revised: Asshown in Figure 2, the response-optimized models can achieve a significantly high accuracyeven when only just 5 % of the training examplesfrom novel subjects ( ∼ 500 samples) are usedto train the readout.

Original: This remarkable generalization of response-optimized networks, butnot task-optimized networks, to novel subjects suggests our models contain appropriate “inductivebiases" that improve sample efficiency.
Revised: This remarkable generalization of response-optimized networks, exceedingthat of task-optimized networks, to novel subjects suggests our models contain appropriate“inductive biases" that improve sample efficiency.

Original: Computational models generalize to OOD datasets We next investigated out-of-distribution (OOD) testsettings to understand the generalization behavior ofresponse-optimized models.
Revised: Computational models generalize to OOD datasets We next investigated out-of-distribution(OOD) test settings to understand the generalization behavior of response-optimized models.

Original: The first OOD setting weconsider is the NSD-synthetic experiment, comprisingfMRI recordings to synthetic stimuli (samples shown in Figure 3) from the same NSD subjects.
Revised: Thefirst OOD setting we consider is the NSD-synthetic experiment, comprising fMRI recordings tosynthetic stimuli (samples shown in Figure 3) from the same NSD subjects.

Original: As shown in Figure 3, response-optimized models significantly outperformtask-optimized models  in  this OOD domain (‘artificial’stimuli), suggesting that the mapping from task-optimized DNN features to voxel responses, was domain/datasetspecific.
Revised: As shown in Figure 3,response-optimized models significantly outperform task-optimized models (except in region VO)in this OOD domain (‘artificial’ stimuli), suggesting that the mapping from task-optimized DNNfeatures to voxel responses, was domain/dataset-specific.

Original: Response-optimized models, on the other hand,allow for flexible OOD generalization.
Revised: Response-optimized models, on the otherhand, allow for flexible OOD generalization.

Original: Generalization toother novel fMRI datasets, including [27, 28, 29] is shownand discussed in the Suppl. Results.
Revised: Generalization to other novel fMRI datasets, including[49, 50, 51] is shown and discussed in the Appendix B.2.

Original: Learned spatial masks reproduce the fine-grainedretinotopic organization of early visual cortex
Revised: Learned spatial masks reproduce the fine-grained retinotopic organization of early visual cortex

Original: Next,we estimate the population receptive field from the learnedspatial masks and compare the estimated parametersagainst the  measurements from the independent  pRF localizer scan session.
Revised: The factorized readout in the proposed model allows us to separate spatial and feature selectivity. We characterized the spatial tuning of early visual areas by estimating the population receptive field(pRF) from the learned spatial masks of the proposed response-optimized model; we compare theestimated parameters against measurements from the independent NSD pRF localizer experimentthat uses artificial stimuli. The pRF is defined as the region of the visual field within which a stimulusresults in increased aggregated activity across a population of neurons, as reflected in fMRI. Basedon the analysis of these data, each brain voxel’s retinotopic organization can be described with threeparameters: the pRF location in polar coordinates (polar angle and eccentricity), and size.

Revised: Usingthe voxel-wise spatial mask of the readout in our encoding model, we estimated the pRF location asthe average of the coordinates weighted by the learned spatial mask value at each grid location.

Revised: Tocompare the estimated eccentricity against the corresponding measurements from pRF localizers,henceforth called ‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient betweenthe measured and predicted eccentricity arrays in each ROI.

Revised: To compare the estimated polar anglevalues against corresponding measurements from pRF localizer scans, henceforth called ‘polar angleagreement’ , we adopted a measure from circular statistics called circular correlation coefficient [52],due to the circular nature of angular data (Appendix A.4).

Original: It is important to note that the encodingmodels do not account for the spatial locations of target voxels.
Revised: It is important to note that the encodingmodels do not account for the spatial locations of target voxels.

Original: Despite this, the estimated spatial topographic organization from the encoding models exhibits remarkable levels of agreementwith the fine-scaled retinotopic organization of the cortex without any explicit supervision to doso (Figure 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angleagreement ∈ 
Revised: Despite this, the estimated spatialtopographic organization from the encoding models exhibits remarkable levels of agreement withthe fine-scaled retinotopic organization of the cortex without any explicit supervision to do so(Figure 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angleagreement ∈ [0.70-0.82].

Original: We further compared this agreement on the estimated pRF parametersfor the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linearreadouts from the fixed convolutional core of the response-optimized model (pre-trained on theinitial set of 4 subjects).
Revised: We further compared this agreement on the estimated pRF parametersfor the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linearreadouts from the fixed convolutional core of the response-optimized model (pre-trained on theinitial set of 4 subjects).

Original: We find that while the agreement increases with the sample size used tofit readouts, even with much fewer samples (e.g. 100), we can still characterize their retinotopicorganization (Figure 4C).
Revised: We find that while the agreement increases with the sample size used tofit readouts, even with much fewer samples (e.g. 100), we can still characterize the retinotopicorganization (Figure 4C).

Revised: This is better than the retinotopic agreement observed for the CORnetS model (which also employed a factorized readout, enabling characterization of spatial tuning(Appendix Figure A.8)).

Original: Previously, retinotopic organization of the human visual cortex has beendominantly studied with spatially modulated stimuli.
Revised: Previously, retinotopic organization of the human visual cortex has beendominantly studied with spatially modulated stimuli.

Original: While encoding models have been shown tobe broadly useful for characterizing retinotopy [30, 31], a precise quantification of the agreementbetween model-estimated and localizer-estimated retinotopic parameters, as well as the effect ofsample sizes on this agreement, has been lacking.
Revised: While encoding models have been shown tobe broadly useful for characterizing retinotopy [53, 31], a precise quantification of the agreementbetween model-estimated and localizer-estimated retinotopic parameters, as well as the effect ofsample sizes on this agreement, has been lacking.

Original: Here, we present an alternate approach basedon naturalistic stimulation and response-optimized encoding models to delineate subject-specificretinotopic maps and precisely quantify the agreement.
Revised: Here, we present an alternate approach basedon naturalistic stimulation and response-optimized encoding models to delineate subject-specificretinotopic maps and precisely quantify the agreement.

Original: Remarkably, the retintopic organizationestimated from the proposed encoding models is even more spatially smooth than the localizer-basedorganization, despite not being informed by anatomy.
Revised: Remarkably, the retintopic organizationestimated from the proposed encoding models is even more spatially smooth than the localizer-basedorganization, despite not being informed by anatomy.

Original: Response-optimized models characterize transformations of image representations along thevisual cortical hierarchy
Revised: Response-optimized models characterize transformations of image representations along thevisual cortical hierarchy

Original: (a) Monotonically increasing alignment with human perception along the ventral visual stream
Revised: (a) Monotonically increasing alignment with human perception along the ventral visual stream

Original: We evaluated models on behavioral data from a Multiple Arrangement task where participantsarranged NSD images according to the similarity of their content.
Revised: We evaluated models on behavioral data from a Multiple Arrangement task where participantsarranged NSD images according to the similarity of their content.

Original: This behavioral analysis of neuralnetwork models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capturehuman perceptual patterns and image semantics than early visual areas and this quantitative agreementincreases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’analysis discussed in Methods).
Revised: This behavioral analysis of neuralnetwork models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capturehuman perceptual patterns and image semantics than early visual areas and this quantitative agreementincreases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’analysis discussed in Methods).

Original: Importantly, responses patterns in their respective models groupsemantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region
Revised: Importantly, responses patterns in their respective models groupsemantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region

Original: VO).
Revised: VO).

Original: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes grouptogether in the left (Figure 5A), despite wide variations in the visual appearance of these categories.
Revised: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes grouptogether in the left (Figure 5A), despite wide variations in the visual appearance of these categories.

Original: This is remarkable, given that the models received no supervision from perceptual/semantic measures.
Revised: This is remarkable, given that the models received no supervision from perceptual/semantic measures.

Original: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Suppl.
Revised: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Appendix Figure

Original: Results.
Revised: A.10.

Original: Thissuggests that these models, solely optimized for high-level visual cortical response prediction –without any semantic supervision – can even serve as powerful models for uncovering the structure of mental representations.
Revised: This suggests that these models, solely optimized for high-level visual cortical responseprediction – without any semantic supervision – can even serve as powerful models for uncoveringthe structure of mental representations.

Original: Intriguingly, the representational structure of predicted responses exhibitsstronger agreement with perceived similarity than the representational structure of measured responses.
Revised: Intriguingly, the representational structure of predictedresponses exhibits stronger agreement with perceived similarity than the representational structureof measured responses.

Original: This suggests that both the perceptual and semantic information are likely more explicit in predictedresponses (even for held-out data) than measured responses.
Revised: This suggests that both the perceptual and semantic information are likelymore explicit in predicted responses (even for held-out data) than measured responses.

Original: One possible explanation is that fMRImeasures of brain activity have undesired noise components and predictive models, in particular,response-optimized models as shown here, can denoise this fMRI activity.
Revised: One possibleexplanation is that fMRI measures of brain activity have undesired noise components and predictivemodels, in particular, response-optimized models as shown here, can denoise this fMRI activity (seealso Appendix Figure A.3).(b)

Original: Monotonically increasing separability of category information along the ventral visualstream Task-optimized computational models have suggested that the ventral stream hierarchy‘untangles’ representations through a sequence of processing steps, so that representations of objectsand categories that are inseparable at the initial processing stage (e.g., V1) become untangled at thelast stage of the hierarchy [32, 10].
Revised: Monotonically increasing separability of category information along the ventral visualstream Task-optimized computational models have suggested that the ventral stream hierarchy‘untangles’ representations through a sequence of processing steps, so that representations of objectsand categories that are inseparable at the initial processing stage (e.g., V1) become untangled atthe last stage of the hierarchy [54, 23].

Original: Here, we present a strong evidence for this increased separabilitythrough hypothesis-agnostic response-optimized models.
Revised: Here, we present a strong evidence for this increasedseparability through hypothesis-agnostic response-optimized models.

Original: Previous studies have shown that therepresentational geometry for the same set of stimuli varies systematically across different corticalareas, providing a useful signature for how information is transformed across different corticalprocessing streams [32].
Revised: Previous studies have shownthat the representational geometry for the same set of stimuli varies systematically across differentcortical areas, providing a useful signature for how information is transformed across differentcortical processing streams [54].

Original: Here, we employ the representational similarity analysis (RSA) framework to relate emergent representations in response-optimized models of difference brain regions againsta ground truth adjacency matrix defined by object category labels.
Revised: Here, we employ the representational similarity analysis (RSA)framework to relate emergent representations in response-optimized models of difference brainregions against a ground truth adjacency matrix defined by object category labels.

Original: Keeping model architecture thesame, simply by changing the response targets for an encoding model from voxels in visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extracted representation.
Revised: Keeping model architecture the same, simply by changing the response targets for an encoding model from voxelsin visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extractedrepresentation.

Original: The increasingly prominent block-diagonal structure in the RSMs along the ventral visual stream(Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlights that thedistributed activity patterns to exemplars of the same category become increasingly more similar; anddistributed patterns to exemplars of different categories become progressively dissimilar along theprocessing stream, strongly supporting the computational goal of the ventral visual stream as a visualcategorization system.
Revised: The increasingly prominent block-diagonal structure in the RSMs along the ventralvisual stream (Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlightsthat the distributed activity patterns to exemplars of the same category become increasingly moresimilar; and patterns to exemplars of different categories become progressively dissimilar alongthe processing stream, strongly supporting the computational goal of the ventral visual stream asa categorization system.

Original: We also quantified the separability of categorical information in differentvisual areas (Figure 5B).
Revised: We also quantified the separability of categorical information in differentvisual areas (Figure 5B).

Original: Here again, we see that this finding regarding increasing separability ofcategorical information holds across both datasets containing different visual categories.
Revised: Here again, we see that this finding regarding increasing separability ofcategorical information holds across both datasets containing different visual categories.

Original: Response-optimized models reveal high-level visual information within ventral-occipital cortex
Revised: Response-optimized models reveal high-level visual information within ventral-occipital cortex

Original: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model andwhether they are capable of discriminating between complex categories.
Revised: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model andwhether they are capable of discriminating between complex categories.

Original: Instead of looking at allmodel neurons emulating voxels, we built a compact neural representational space by passing a largestimulus set ( ∼ 27,000 images from the THINGS database [20]) through the predictive models andthen performing Principal Components Analysis (PCA) on the predicted responses of this network(see Suppl. Methods).
Revised: Instead of looking at allmodel neurons emulating voxels, we built a compact neural representational space by passing a largestimulus set ( ∼ 27,000 images from the THINGS database [44]) through the predictive models andthen performing Principal Components Analysis (PCA) on the predicted responses of this network(Appendix A.6).

Original: Visualizing the images from top concept categories that elicit highest andlowest activation of individual principal components (PCs) reveals complex stimulus dimensions: thefirst principal component, which explained ∼ 49% of the variance in responses strongly separatesfaces, animals and bodies from inanimate objects, despite vast variations in the visual appearanceof images within sub-categories.
Revised: Visualizing the images from top concept categories that elicit highest and lowestactivation of individual principal components (PCs) reveals complex stimulus dimensions: the firstprincipal component, which explained ∼ 49% of the variance in responses strongly separates faces,animals and bodies from inanimate objects, despite vast variations in the visual appearance ofimages within sub-categories.

Original: The second principal component, accounting for ∼ 16% variance,distinguished objects based on their shape (circular versus rectilinear objects), suggesting that suchshape features are important for VO.
Revised: The second principal component, accounting for ∼ 16% variance,distinguished objects based on their shape (circular versus rectilinear objects), suggesting that suchshape features are important for VO.

Original: Importantly, these features do not emerge when the sameanalysis is applied to early visual ROIs (V1-V4).
Revised: Importantly, these features do not emerge when the sameanalysis is applied to early visual ROIs (V1-V4).

Original: Response-optimized models exhibit a shape bias A striking known discrepancy between humanand machine vision is the shape vs. texture bias: human classification decisions are shape-biasedwhereas CNNs trained for CV tasks tend to classify images by texture [24, 33].
Revised: Response-optimized models exhibita shape bias A striking known discrepancy between human and machine vision is the shape vs. texture bias: human classification decisions are shape-biased whereas CNNstrained for CV tasks tend to classifyimages by texture [55, 56].

Original: We study whetherthe shape bias of humans is reflected in response-optimized models of cortical regions that subservevisual categorization in humans.
Revised: We studywhether the shape bias of humans isreflected in response-optimized models of cortical regions that subservevisual categorization in humans.

Revised: Wefirst train a linear layer on top ofthe predicted responses of responseoptimized models to map these responses to visual categories in the

Original: First, we assess the transfer performance of response-optimizedmodels for classification on held-out stimuli from the ImageNet-16 dataset as well as the Silhouettedataset described in [24] (see Suppl. Methods).
Revised: ImageNet-16 database [55] using asmall subset of this database. We thenassess the transfer performance ofresponse-optimized models for classification on held-out stimuli from the ImageNet-16 dataset as well as the Silhouette dataset described in [55](Appendix A.10).

Original: We find that the classification performanceis significantly better than what would be expected from the same network architecture with arandom initialization (Figure 6B), suggesting that a brain response-optimization objective alonecan yield effective representations.
Revised: We find that theclassification performance is significantly better than what would beexpected from the same network architecture with a random initialization (Figure 6B), suggesting that thebrain response-optimization objectiveyields effective representations.

Original: Further, this classification performance is better using modelsof intermediate and high-level visual areas (V4, LO and VO) than early visual areas, providingfurther support for the separability of categorical information in high-level, but not early visualcortex [7].
Revised: Further, this classification performance isbetter using models of intermediate and high-level visual areas (V4, LO and VO) than early visual ar- eas, even approaching the transfer performance of an MS-COCO-trained object classification networkwith the same architecture (Figure 6B,top, see Appendix A.10 for model details), providing furthersupport for the separability of categorical information in high-level, but not early visual cortex [11].

Original: The LO model has the highest classification performance on the Silhouette dataset (whichsolely contain shape information), consistent with and providing further evidence for the previoushypotheses that LO tracks object shape information [34].
Revised: The LO model has the highest classification performance on the Silhouette dataset (which solelycontain shape information), consistent with and providing further evidence for previous hypothesesthat LO tracks object shape information [57].

Original: We then evaluate the  performance of thismapped response-optimized model on the diagnostic  cue conflict dataset  to assess whether the modelmakes its categorization decisions based on shape or texture.
Revised: We then evaluate the classification performance of thismapped response-optimized model on the diagnostic texture-shape cue conflict dataset [55] to assesswhether the model makes its categorization decisions based on shape or texture (Appendix A.10).

Original: We follow [24] and define shape biasas the percentage of times the model classifies an image based on its shape over texture, provided itreturned either the correct shape or correct texture label for the image.
Revised: Wefollow [55] and define shape bias as the percentage of times the model classifies an image based onits shape over texture, provided it returned either the correct shape or correct texture label for theimage.

Original: A model is shape-biased if itsshape bias > 50 % .
Revised: A model is shape-biased if its shape bias > 50 % .

Original: We observed that all response-optimized models exhibit a shape bias (Figure 6B[bottom]), with this metric ranging from 56.7-69.4 % , suggesting that representations within theventral visual stream are likely biased towards shapes than textures.
Revised: We observed that all response-optimizedmodels exhibit a shape bias (Figure 6B [bottom], Appendix Figure A.1), with this metric rangingfrom 56.7-69.4 % , suggesting that representations within the ventral visual stream are likely biasedtowards shapes than textures.

Original: This is in stark contrast tostandard CV models which show a  texture bias.
Revised: This is in stark contrast to standard CV models which show a texturebias, for e.g., the same architecture trained for multi-label object classification on MS-COCO shows astrong texture bias and performs significantly worse on the Silhouettes dataset (Figure 6B, gray bar),suggesting that the distribution of NSD stimuli (which are drawn from MS-COCO) is not sufficientto drive the shape bias and the training objective (matching to neural responses rather than objectcategory labels) matters.

Original: Further, as shown in Figure 6B (bottom), it isnot the  mapping to neural response but the way features are optimized that explains the differingshape/texture bias of task-optimized and response-optimized  models.
Revised: Further, as shown in Figure 6B (bottom), it is not the linear mapping toneural response but the way features are optimized that explains the differing shape/texture bias oftask-optimized and response-optimized encoding models.

Original: We also visualized the features in different response-optimized models using an image synthesisprocedure (described in  Methods).
Revised: We next performed a feature visualizationanalysis to find visual patterns that would maximally activate individual model neurons emulatingbrain voxels in different response-optimized encoding models. Following [58], we start with arandom noise input x 0 for each model neuron i and iteratively update the input along the gradient∂a i ∂x to synthesize inputs that would result in higher and higher predicted activation a i for that neuron(see Suppl. Methods).

Original: Visual inspection shows that the features increase in complexityas we go from early to high-level visual ROI models: simple edge-like features or high-low frequencydetectors are seen in V1, curves and edges are seen in V2, more complex curvatures in V4 and finally,fully-formed complex shapes like concentric hexagons are seen in LO and VO (Figure 6A).
Revised: Visual inspection shows that the features increase in complexity as we go fromearly to high-level visual ROI models: simple edge-like features or high-low frequency detectors areseen in V1, curves and edges are seen in V2, more complex curvatures in V4 and finally, fully-formedcomplex shapes like concentric hexagons are seen in LO and VO (Figure 6A, Appendix Figure A.7).

Revised: We also validated this increased complexity using quantitative measures based on compression ratios,as shown in Appendix Table A.3.

Original:  Discussion
Revised:  Discussion

Original: Here, we set out to answer the question: How far can deep learning on neural responses alone get usin predicting neural responses to ID and OOD test sets and understanding the ventral visual pathway?

Original: Thus, far, visual neuroscience had been largely measurement-limited, constraining modeling effortsthat bring neuroscience data to bear on the model development process.

Original: The large sample size of NaturalScenes Dataset presented an opportunity to change this status quo.
Revised: In this study, we leveraged the large sample size and natural variation represented in the Natural

Original: We capitalized on thenatural variation in NSD to train deep neural networks solely with supervision from brain responseprediction.
Revised: Scenes Dataset to train deep neural networks directly on the brain response.

Original: We studied model generalization in different settings: from synthetic stimuli to novel fMRIdatasets with widely different preprocessing pipelines.
Revised: We studied modelgeneralization in various settings: from synthetic stimuli to novel fMRI datasets with differentpreprocessing.

Original: We demonstrated success in recapitulatingkey features of early visual areas (e.g. their precise retinotopic organization), representationaltransformations along the ventral visual stream along with demonstrating that high-level visualfeatures like faces, places and complex shapes are represented in high-level visual areas.
Revised: Our results recapitulated key features of early visual areas (e.g. their precise retinotopicorganization), representational properties in higher level areas.

Original: Remarkably,we also demonstrated that despite no supervision from human behavioral measures, these modelscan capture human behavioral abilities, including not only object categorization, but also otherfine-grained measures such as perceived image similarity.
Revised: We also demonstrated that responseoptimized models can capture human behavior, including not only object categorization, but also otherfine-grained measures such as perceived image similarity.

Original: These hypothesis-agnostic models, webelieve, can complement task-optimized models in furthering our understanding of human vision; theresponse-optimization approach, in principle, allows exploring infinite paths in the model space, andwhen optimized properly, converging towards one that explains maximum possible variance in thedata.
Revised: The response-optimization approach, inprinciple, allows exploring infinite paths in the model space, and when optimized properly, convergingtowards one that explains maximum possible variance in the data.

Original: Since any features that emerge in these optimized networks are a result of optimization to matchneural targets, unconfounded by any other supervision signal, these models can provide stronger testsabout the visual representations within different visual areas.
Revised: Since any features that emergein these optimized networks are a result of optimization to match neural targets, unconfounded byany other supervision signal, these models can provide stronger tests about the visual representationswithin different visual areas.

Original: In terms of human object categorization behavior (as tested on ImageNet-16), our models lacksignificantly compared to ImageNet-supervised DNN models.
Revised: In terms of human object categorization behavior, our models lack significantly compared to ImageNet-supervised DNN models.

Original: While part of this may be due to thedomain shift (NSD/COCO → ImageNet), the restricted transfer learning setting (a single linear layerto map predicted voxel responses to category labels), small size of the transfer set or simply the coarsespatial resolution of fMRI, this gap isstill worth exploring.In this study, we focused on the use of
Revised: While part of this may be due to the domain shift, the restricted transfer learning setting (a single linear layer mapping), small size of the transfer set orsimply the coarse spatial resolution of fMRI, this gap is worth exploring.

Original: DNN models in understanding the ventral visual stream but using neural data to build better modelsfor other communities (e.g. computer vision) is also scientifically fascinating and has the potential todevelop reinforcing cycles.
Revised: Further, using neural data tobuild better models for other communities (e.g. computer vision) is another scientifically fascinatingdirection and can develop reinforcing cycles.

Original: It will be interesting to see whether regularizing object categorizationmodels with model-predicted neural responses can grant them brain-like biases (e.g. shape bias)and capabilities (e.g. OOD generalization).

Original: Studies have shown that explicitly encouraging neuralnetworks to build representations that are like neural representations or principles in V1  can improvetheir robustness under stimulus noise and adversarial attacks [35, 36].
Revised: Studies have shown that explicitly encouraging neuralnetworks to build representations similar V1 representations can improve their robustness undernoise and adversarial attacks [59, 60].

Original: In a similar vein, regularizationwith high-level visual cortical activity might help  close the gap between human and machine vision,since this activity is the neural basis for human visual intelligence.
Revised: In a similar vein, regularization with high-level visual corticalactivity might help grant models brain-like biases (e.g. shape bias) and close the gap between humanand machine vision. Checklist For all authors...

Original: A Appendix
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] They are described within the

Revised: Discussion section

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A] We don’t

Revised: foresee any negative societal impacts of this work.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Revised: them?

Revised: [Yes] I’ve read all guidelines

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A]

Revised: No theoretical

Revised: results are discussed in this paper.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [N/A]

Revised: No theoretical results

Revised: are discussed in this paper.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Revised: mental results (either in the supplemental material or as a URL)?

Revised: [Yes] Yes, the code is

Revised: included in the Supplementary Material.

Revised: (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters, how they were

Revised: chosen)?

Revised: [Yes] Yes, all training details are included in the Methdos and Supplementary

Revised: Methods.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [Yes] Yes, they are displayed in all figures.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Revised: of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] Yes.

Revised: All models were trained,

Revised: validated and analysed on an NVIDIA Titan Xp GPU.

Revised: Training each response-optimized

Revised: model on 1 GPU took ∼ 8 hours.

Revised: Additional details are listed in the Supplementary.

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] e.g. [47, 61]

Revised: (b) Did you mention the license of the assets?

Revised: [No] The assets employed in this study were

Revised: made publicly available by the creators

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [No]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [No] The assets employed in this study were made publicly available

Revised: by the creators

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: information or offensive content?

Revised: [No] The assets don’t contain any offensive content

Revised: to our knowledge.

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Revised: [N/A](b)

Revised: Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Revised: [N/A]