Original: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models
Revised: Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding Models

Original: Abstract
Revised: Abstract

Original: Decades of experimental research based on simple, abstract stimuli has revealed the coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complex visual categories in the anterior ventral stream.
Revised: Decades of experimental research based on simple, abstract stimuli has revealed the coding principles of the ventral visual processing hierarchy, from the presence of edge detectors in the primary visual cortex to the selectivity for complex visual categories in the anterior ventral stream.

Original: However, these studies are, by construction, constrained by their a priori hypotheses.
Revised: However, these studies are, by construction, constrained by their a priori hypotheses.

Original: Furthermore, beyond the early stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.
Revised: Furthermore, beyond the early stages, precise neuronal tuning properties and representational transformations along the ventral visual pathway remain poorly understood.

Original: In this work, we propose to employ response-optimized encoding models trained solely to predict the functional MRI activation, in order to gain insights into the tuning properties and representational transformations in the series of areas along the ventral visual pathway.
Revised: In this work, we propose to employ response-optimized encoding models trained solely to predict the functional MRI activation, in order to gain insights into the tuning properties and representational transformations in the series of areas along the ventral visual pathway.

Original: We demonstrate the strong generalization abilities of these models on artificial stimuli and novel datasets.
Revised: We demonstrate the strong generalization abilities of these models on artificial stimuli and novel datasets.

Original: Intriguingly, we find that response-optimized models trained towards the ventral-occipital and lateral-occipital areas, but not early visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.
Revised: Intriguingly, we find that response-optimized models trained towards the ventral-occipital and lateral-occipital areas, but not early visual areas, can recapitulate complex visual behaviors like object categorization and perceived image-similarity in humans.

Original: We further probe the trained networks to reveal representational biases in different visual areas and generate experimentally testable hypotheses.
Revised: We further probe the trained networks to reveal representational biases in different visual areas and generate experimentally testable hypotheses.

Original:  Introduction
Revised:  Introduction

Original: Understanding the nature of representations and computations in the visual system has been a longstanding goal in neuroscience.
Revised: Understanding the nature of representations and computations in the visual system has been a longstanding goal in neuroscience.

Original: Substantial progress been made by presenting model organisms with simple, abstract stimuli like noise or sine-wave gratings and studying the evoked responses.
Revised: Substantial progress been made by presenting model organisms with simple, abstract stimuli like noise or sine-wave gratings and studying the evoked responses.

Original: These carefully crafted experiments, for instance, revealed the presence of edge detectors in the primary visual cortex [1] and sparked the search for the ‘optimal input’ beyond early visual areas across the visual cortical hierarchy.
Revised: These carefully crafted experiments, for instance, revealed the presence of edge detectors in the primary visual cortex [1] and sparked the search for the ‘optimal input’ beyond early visual areas across the visual cortical hierarchy.

Original: Significant efforts have also been made to understand the feature complexity of tuning in V4 and IT neurons [2, 3, 4, 5].
Revised: Significant efforts have also been made to understand the feature complexity of tuning in V4 and IT neurons [2, 3, 4, 5].

Original: Over the last two decades, through numerous experiments, recording and analysis techniques, a conceptual understanding has emerged that early visual areas extract low-level features like edges or curves, mid-level regions extract complex local shapes, and high-level regions encode semantic categories, like faces or scenes [6, 7, 8, 9, 10].
Revised: Over the last two decades, through numerous experiments, recording and analysis techniques, a conceptual understanding has emerged that early visual areas extract low-level features like edges or curves, mid-level regions extract complex local shapes, and high-level regions encode semantic categories, like faces or scenes [6, 7, 8, 9, 10].

Original: While much of the details remain unknown, there is accumulating evidence that visual categorization occurs in the ventral temporal cortex in humans [11].
Revised: While much of the details remain unknown, there is accumulating evidence that visual categorization occurs in the ventral temporal cortex in humans [11].

Original: This understanding was enabled by synthesizing findings across different hypothesis-driven studies, most of them employing artificial stimuli atypical of real-world vision, thus lacking ethological validity.
Revised: This understanding was enabled by synthesizing findings across different hypothesis-driven studies, most of them employing artificial stimuli atypical of real-world vision, thus lacking ethological validity.

Original: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks, have emerged as the most accurate models of the primate ventral visual stream [12, 13, 14, 15].
Revised: Recently, deep convolutional neural networks (CNNs) optimized for computer vision (CV) tasks, have emerged as the most accurate models of the primate ventral visual stream [12, 13, 14, 15].

Original: Furthermore, these CV task-optimized models offer a hierarchical correspondence to visual areas along the ventral stream: early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [12, 16, 17, 18, 19, 20, 21, 22].
Revised: Furthermore, these CV task-optimized models offer a hierarchical correspondence to visual areas along the ventral stream: early CNN layers best predict V1, while intermediate and late layers best predict V4 and IT [12, 16, 17, 18, 19, 20, 21, 22].

Original: They provide intelligibility in the form of the goal-driven perspective, highlighting the functional role of representations [23].
Revised: They provide intelligibility in the form of the goal-driven perspective, highlighting the functional role of representations [23].

Original: However, we are far from a complete understanding of the top-down computational objectives (the battery of ethologically-relevant tasks) that may have shaped neural representations through evolution or experience-based learning; in the absence of right hypotheses, the task-optimized approach can fall short.
Revised: However, we are far from a complete understanding of the top-down computational objectives (the battery of ethologically-relevant tasks) that may have shaped neural representations through evolution or experience-based learning; in the absence of right hypotheses, the task-optimized approach can fall short.

Original: Studies have already revealed significant inconsistencies between these models and human behavior and gaps remain in their respective robustness to distortions, image-level consistency etc [24].
Revised: Studies have already revealed significant inconsistencies between these models and human behavior and gaps remain in their respective robustness to distortions, image-level consistency etc [24].

Original: Recently, the idea of using neural data in the model development process has gained some traction, and demonstrated success in increasing both model-human behavior [25, 26, 27], and model-brain alignment, while revealing novel insights about information processing in neural systems [28, 29, 30, 31].
Revised: Recently, the idea of using neural data in the model development process has gained some traction, and demonstrated success in increasing both model-human behavior [25, 26, 27], and model-brain alignment, while revealing novel insights about information processing in neural systems [28, 29, 30, 31 ? ].

Original: In this study, we leverage recent advances in large-scale fMRI data collection [32] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.
Revised: In this study, we leverage recent advances in large-scale fMRI data collection [32] to train hypothesisagnostic computational models directly on neural data with novel predictive precision.

Original: Importantly, the stimulus set employed in this study contains crowded images of multiple objects in their natural contexts, thus being more typical of everyday scenes and allowing us to characterize neural representations and computations in rich, naturalistic conditions.
Revised: Importantly, the stimulus set employed in this study contains crowded images of multiple objects in their natural contexts, thus being more typical of everyday scenes and allowing us to characterize neural representations and computations in rich, naturalistic conditions.

Original: We demonstrate the remarkable ability of these “response-optimized” neural encoding models in reproducing the stimulus-response relationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassing the unprecedented prediction performance of task-optimized models.
Revised: We demonstrate the remarkable ability of these “response-optimized” neural encoding models in reproducing the stimulus-response relationship in voxels along the ventral visual stream for arbitrary images, equaling or surpassing the unprecedented prediction performance of task-optimized models.

Original: We show that these models generalize well to contexts that are vastly different from their training conditions (artificial stimuli, novel subjects, etc.), providing a strong quantitative validation for the response-optimization approach of building encoding models for neurobiological visual systems beyond the retina [33].
Revised: We show that these models generalize well to contexts that are vastly different from their training conditions (artificial stimuli, novel subjects, etc.), providing a strong quantitative validation for the response-optimization approach of building encoding models for neurobiological visual systems beyond the retina [33].

Original: Our approach further goes beyond prediction accuracy and aims to gain insights about neuronal tuning properties.
Revised: Our approach further goes beyond prediction accuracy and aims to gain insights about neuronal tuning properties.

Original: Specifically, we implement neural encoding models that disentangle the processing of the “what” and the “where.”
Revised: Specifically, we implement neural encoding models that disentangle the processing of the “what” and the “where.”

Original: This allows us to reveal the selectivity of individual brain voxels along both location and appearance dimensions.
Revised: This allows us to reveal the selectivity of individual brain voxels along both location and appearance dimensions.

Original: We first demonstrate that the estimated spatial topographic organization from our neural encoding models exhibits remarkable agreement with the fine-scaled retinotopic organization of the cortex, which is typically measured with dedicated, painstaking experimental paradigms.
Revised: We first demonstrate that the estimated spatial topographic organization from our neural encoding models exhibits remarkable agreement with the fine-scaled retinotopic organization of the cortex, which is typically measured with dedicated, painstaking experimental paradigms.

Original: We analyze the features that emerge in the neural encoding models to understand the tuning properties of cortical voxels.
Revised: We analyze the features that emerge in the neural encoding models to understand the tuning properties of cortical voxels.

Original: This analysis suggests shape-biased tuning across the ventral visual stream, increasing in complexity from Gabor-like features in V1, curves/angles in V2 to concentric and complex shape processing in V4 and higher-order lateral-occipital and ventral-occipital ROIs respectively.
Revised: This analysis suggests shape-biased tuning across the ventral visual stream, increasing in complexity from Gabor-like features in V1, curves/angles in V2 to concentric and complex shape processing in V4 and higher-order lateral-occipital and ventral-occipital ROIs respectively.

Original: We also go beyond single-voxel analysis to characterize how representations evolve along the ventral visual pathway.
Revised: We also go beyond single-voxel analysis to characterize how representations evolve along the ventral visual pathway.

Original: Probing the information encoded in model-predicted patterns of activity reveals the monotonically increasing separability of category information along the ventral stream hierarchy, strongly supporting the computational goal of the ventral visual stream as a visual categorization system.
Revised: Probing the information encoded in model-predicted patterns of activity reveals the monotonically increasing separability of category information along the ventral stream hierarchy, strongly supporting the computational goal of the ventral visual stream as a visual categorization system.

Original: We further demonstrate the correspondence between model representations (of high-level visual areas) and mental representations (as assessed with human similarity judgements), suggesting that these response-optimized models inherently capture human psychological judgements, without any explicit supervision to do so.
Revised: We further demonstrate the correspondence between model representations (of high-level visual areas) and mental representations (as assessed with human similarity judgements), suggesting that these response-optimized models inherently capture human psychological judgements, without any explicit supervision to do so.

Original: Furthermore, a principle components analysis reveals that models of high-level visual ROIs capture complex stimulus dimensions like faces, bodies and shapes.
Revised: Furthermore, a principle components analysis reveals that models of high-level visual ROIs capture complex stimulus dimensions like faces, bodies and shapes.

Original: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, naturalistic datasets can be used to confirm neural phenomena characterized painstakingly using controlled experimental paradigms that might be hard to replicate in individual subjects.
Revised: Our paper illustrates how neural encoding models trained on hypothesis-agnostic, naturalistic datasets can be used to confirm neural phenomena characterized painstakingly using controlled experimental paradigms that might be hard to replicate in individual subjects.

Original: More importantly, however, we show how in silico experiments on neural encoding models can generate novel hypotheses and provide a powerful framework for building broad, generalizable theories about neural information processing.
Revised: More importantly, however, we show how in silico experiments on neural encoding models can generate novel hypotheses and provide a powerful framework for building broad, generalizable theories about neural information processing.

Original:  Materials and Methods
Revised:  Materials and Methods

Original: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is provided elsewhere [32] (see also the Appendix).
Revised: Natural Scenes Dataset A detailed description of the Natural Scenes Dataset (NSD 1 ) is provided elsewhere [32] (see also the Appendix).

Original: The dataset contains measurements of fMRI responses from 8 participants who each viewed 9,000–10,000 natural scenes.
Revised: The dataset contains measurements of fMRI responses from 8 participants who each viewed 9,000–10,000 natural scenes.

Original: A special set of 1,000 images were shared across subjects; the remaining were mutually exclusive.
Revised: A special set of 1,000 images were shared across subjects; the remaining were mutually exclusive.

Original: We focused on modeling responses within 5 visual cortical ROIs in the study.
Revised: We focused on modeling responses within 5 visual cortical ROIs in the study.

Original: Three ROIs belonging to the retinoptic early visual cortex, namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session [32].
Revised: Three ROIs belonging to the retinoptic early visual cortex, namely, V1v, V2v and hV4 were defined using a population receptive field (pRF) localizer scan session [32].

Original: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipital areas (LO1-2) were delineated using a visual probabilistic atlas [34](Figure 1B).
Revised: Two higher order ROIs, namely ventral-occipital areas (VO1-2) and lateral-occipital areas (LO1-2) were delineated using a visual probabilistic atlas [34](Figure 1B).

Original: Response-optimized neural encoding model
Revised: Response-optimized neural encoding model

Original: We trained separate voxel-level neural encoding models for each of the above regions with the same backbone architecture.
Revised: We trained separate voxel-level neural encoding models for each of the above regions with the same backbone architecture.

Original: Keeping the architecture constant across visual areas ensured that differences in emergent representations are driven mainly by the nature of response patterns.
Revised: Keeping the architecture constant across visual areas ensured that differences in emergent representations are driven mainly by the nature of response patterns.

Original: The predictive model comprises a convolutional neural network (CNN) feature extractor shared across all subjects, but unique to specific visual areas.
Revised: The predictive model comprises a convolutional neural network (CNN) feature extractor shared across all subjects, but unique to specific visual areas.

Original: We employ a linear readout model on top of the feature space to predict the responses of individual voxels in a specific region.
Revised: We employ a linear readout model on top of the feature space to predict the responses of individual voxels in a specific region.

Original: The linear readout is factorized into spatial and feature dimensions following [35] (Figure 1A).
Revised: The linear readout is factorized into spatial and feature dimensions following [35] (Figure 1A).

Original: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).
Revised: This allows us to separate spatial tuning (the “where”) from feature tuning (the “what”).

Original: The shared CNN feature extractor consists of four convolutional blocks, with each block comprising the following feedforward computations: two 3 × 3 convolutional layers, each followed by an inner batch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.
Revised: The shared CNN feature extractor consists of four convolutional blocks, with each block comprising the following feedforward computations: two 3 × 3 convolutional layers, each followed by an inner batch norm, and a ReLU; and an anti-aliased AvgPool operation (stride = 2) at the end.

Original: We employ
Revised: We employ

Original: E(2)-steerable convolutions in all our models to extract features invariant to orientation [36].
Revised: E(2)-steerable convolutions in all our models to extract features invariant to orientation [36, 37].

Original: This modeling choice is inspired by neural computations in early visual areas where groups of neurons are known to perform similar computations, e.g., edge or curve detection, at different orientations.
Revised: This modeling choice is inspired by neural computations in early visual areas where groups of neurons are known to perform similar computations, e.g., edge or curve detection, at different orientations.

Original: Each convolutional layer contains 48 feature sets extracted at 8 orientations.
Revised: Each convolutional layer contains 48 feature sets extracted at 8 orientations.

Original: Weights of the readout are a computed as outer products between a spatial filter and a feature vector.
Revised: Weights of the readout are a computed as outer products between a spatial filter and a feature vector.

Original: The spatial filter further has a positivity constraint (enforced using rectification) and is length-normalized for each voxel.
Revised: The spatial filter further has a positivity constraint (enforced using rectification) and is length-normalized for each voxel.

Original: Training and testing response-optimized encoding models We used 4 NSD subjects for training and reserved the other 4 subjects for studying generalization.
Revised: Training and testing response-optimized encoding models We used 4 NSD subjects for training and reserved the other 4 subjects for studying generalization.

Original: The first group collectively saw 37, natural scene images, including the 1,000 shared images.
Revised: The first group collectively saw 37, natural scene images, including the 1,000 shared images.

Original: We used the 1,000 shared images for testing our models and split the remaining stimulus set into 35,000 training and 2,000 validation images.
Revised: We used the 1,000 shared images for testing our models and split the remaining stimulus set into 35,000 training and 2,000 validation images.

Original: All parameters of the neural encoding model were optimized jointly to minimize the masked mean squared error between the predicted and measured fMRI response.
Revised: All parameters of the neural encoding model were optimized jointly to minimize the masked mean squared error between the predicted and measured fMRI response.

Original: This loss allows us to propagate errors through the shared CNN even if the subjects are not exposed to common stimuli since we can exclude the subjects/voxels for which the target response is not present from the loss calculation.
Revised: This loss allows us to propagate errors through the shared CNN even if the subjects are not exposed to common stimuli since we can exclude the subjects/voxels for which the target response is not present from the loss calculation.

Original: We quantify ‘predictive accuracy’ on the test images as the Pearson’s correlation coefficient between the predicted and measured fMRI response at each voxel.
Revised: We quantify ‘predictive accuracy’ on the test images as the Pearson’s correlation coefficient between the predicted and measured fMRI response at each voxel.

Original: We also compare this against the noise-ceiling, expressed as the signal-to-noise ratio when considering the response variability of each voxel across 3 repetitions per image (see Appendix A.5).
Revised: We also compare this against the noise-ceiling, expressed as the signal-to-noise ratio when considering the response variability of each voxel across 3 repetitions per image (see Appendix A.5).

Original: Table A.1 provides a summary of all analyses methods and baselines used for evaluating and interpreting the proposed response-optimized models.
Revised: Table A.1 provides a summary of all analyses methods and baselines used for evaluating and interpreting the proposed response-optimized models.

Original: Baseline models (a) Category ideal observer model: We fit a non-image-computable model using human-generated annotations for NSD images (obtained from the MS-COCO database).
Revised: Baseline models (a) Category ideal observer model: We fit a non-image-computable model using human-generated annotations for NSD images (obtained from the MS-COCO database).

Original: The input to the categorical model is a k-hot encoding vector corresponding to the 80 object categories annotated in the database, where each element indicates the presence/absence of the corresponding category in the image.
Revised: The input to the categorical model is a k-hot encoding vector corresponding to the 80 object categories annotated in the database, where each element indicates the presence/absence of the corresponding category in the image.

Original: We fitted l 2 regularized linear regression models on this space to predict voxel responses for held-out stimuli (Appendix A.8).
Revised: We fitted l 2 regularized linear regression models on this space to predict voxel responses for held-out stimuli (Appendix A.8).

Original: (b) Task-optimized Models: We investigated several DNN models, including AlexNet[37], ResNet-50 [38],  DenseNet [39] and CORnet-s [40], optimized for object recognition on the large-scale ImageNet dataset [41] (Appendix A.3, Table
Revised: (b) Task-optimized Models: We investigated several DNN models, including AlexNet [38], ResNet-50 [39], DenseNet [40] and CORnet-s [41], optimized for object recognition on the large-scale ImageNet dataset [42] (Appendix A.3, Table

Original: A.2).
Revised: A.2).

Original: Since these models can demonstrate lower prediction performance due to the domain shift between ImageNet and NSD stimuli, we also employed a ResNet-50 architecture pre-trained for object detection on the MS-COCO dataset [42] as an additional baseline (since NSD stimuli were drawn from the MS-COCO dataset).
Revised: Since these models can demonstrate lower prediction performance due to the domain shift between ImageNet and NSD stimuli, we also employed a ResNet-50 architecture pre-trained for object detection on the MS-COCO dataset [43] as an additional baseline (since NSD stimuli were drawn from the MS-COCO dataset).

Original: For CORnet-S, we pre-select the layer homologous to each visual region and employ a spatial × feature factorized readout (same as the response-optimized model) on top of the fixed feature backbone, allowing a more direct comparison to the proposed model.
Revised: For CORnet-S, we pre-select the layer homologous to each visual region and employ a spatial × feature factorized readout (same as the response-optimized model) on top of the fixed feature backbone, allowing a more direct comparison to the proposed model.

Original: For the remaining models, we followed the more standard procedure for fitting task-optimized models to voxel responses based on a full linear readout.
Revised: For the remaining models, we followed the more standard procedure for fitting task-optimized models to voxel responses based on a full linear readout.

Original: We extracted features from pre-trained layers of each network and fitted l 2 regularized linear regression models separately for each layer.
Revised: We extracted features from pre-trained layers of each network and fitted l 2 regularized linear regression models separately for each layer.

Original: The regularization parameter was optimized independently for each subject, each layer, each model and for voxels in each visual area by testing among 8 log-spaced values in [1e-4, 1e4].
Revised: The regularization parameter was optimized independently for each subject, each layer, each model and for voxels in each visual area by testing among 8 log-spaced values in [1e-4, 1e4].

Original: For each visual area and model, we identified the layer that best predicts responses on the validation set.
Revised: For each visual area and model, we identified the layer that best predicts responses on the validation set.

Original: These model layers were used to quantify test performance.
Revised: These model layers were used to quantify test performance.

Original: (c) Response-optimized (no rotation symmetries):
Revised: (c) Response-optimized (no rotation symmetries):

Original: Here, we train response-optimized models with the same architecture as the proposed models but without sharing weights across filter orientations.
Revised: Here, we train response-optimized models with the same architecture as the proposed models but without sharing weights across filter orientations.

Original: Characterizing neural representations by behavioral comparisons: (a) Assessing separability of category and semantic information along the visual hierarchy To probe the learned featural tuning in the response-optimized model for each visual area, we used representational similarity analysis (RSA) [43].
Revised: Characterizing neural representations by behavioral comparisons: (a) Assessing separability of category and semantic information along the visual hierarchy To probe the learned featural tuning in the response-optimized model for each visual area, we used representational similarity analysis (RSA) [44].

Original: Here, we correlated the patterns of predicted activity between multiple exemplars from different categories of stimuli to obtain a representational similarity matrix (RSM) for each visual area.
Revised: Here, we correlated the patterns of predicted activity between multiple exemplars from different categories of stimuli to obtain a representational similarity matrix (RSM) for each visual area.

Original: Further, we quantified the separability of categorical information within these similarity matrices by computing a correlation coefficient (Kendall’s τ ) between model RSMs against a ground truth adjacency matrix defined by object category labels.
Revised: Further, we quantified the separability of categorical information within these similarity matrices by computing a correlation coefficient (Kendall’s τ ) between model RSMs against a ground truth adjacency matrix defined by object category labels.

Original: Specifically, the elements of this matrix are 1 if the corresponding two images belong to different categories and 0 if the images belong to the same category.
Revised: Specifically, the elements of this matrix are 1 if the corresponding two images belong to different categories and 0 if the images belong to the same category.

Original: This analysis was first performed using a subset of 104 images from the THINGS database [44] that belong to a pre-defined set of categories for ease of visualization.
Revised: This analysis was first performed using a subset of 104 images from the THINGS database [45] that belong to a pre-defined set of categories for ease of visualization.

Original: These categories were chosen in accordance with previous fMRI studies employing RSA [45, 46] and include the following: { face, hand, elephant, cat, plant, fruit, car, tool } .
Revised: These categories were chosen in accordance with previous fMRI studies employing RSA [46, 47] and include the following: { face, hand, elephant, cat, plant, fruit, car, tool } .

Original: Subsequently, we also validated the obtained results using another large-scale vision dataset, namely ImageNet-16 [47].
Revised: Subsequently, we also validated the obtained results using another large-scale vision dataset, namely ImageNet-16 [48].

Original: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images and computed the similarity between model RSMs and the categorical adjacency matrix following the same approach described for the THINGS dataset.
Revised: We sampled 500 stimuli from each ImageNet-16 category, resulting in a total of 8,000 images and computed the similarity between model RSMs and the categorical adjacency matrix following the same approach described for the THINGS dataset.

Original: To assess visual information flow in relation to high-level semantic understanding, we computed pairwise similarity between predicted responses for held-out NSD images to construct model-predicted RSMs for each visual area.
Revised: To assess visual information flow in relation to high-level semantic understanding, we computed pairwise similarity between predicted responses for held-out NSD images to construct model-predicted RSMs for each visual area.

Original: We constructed a semantic RSM by correlating the 80-D binary category vectors (which indicate whether each category was present in the image or not) for every pair of these NSD images and quantified agreement with model-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).
Revised: We constructed a semantic RSM by correlating the 80-D binary category vectors (which indicate whether each category was present in the image or not) for every pair of these NSD images and quantified agreement with model-predicted RSMs using Kendall’s τ (henceforth, termed the ‘NSD: Semantic’ analysis).

Original: We computed this agreement using measured RSMs for all visual areas as well.
Revised: We computed this agreement using measured RSMs for all visual areas as well.

Original: In RSA described here and anywhere else, error bars (95 % CI) are computed by performing bootstrap sampling over voxels.
Revised: In RSA described here and anywhere else, error bars (95 % CI) are computed by performing bootstrap sampling over voxels.

Original: (b) Assessing alignment with human perception We compared model representations against behavioral data from a Multiple Arrangement task where participants interpreted 100 NSD images and arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ was released with NSD [32]).
Revised: (b) Assessing alignment with human perception We compared model representations against behavioral data from a Multiple Arrangement task where participants interpreted 100 NSD images and arranged them according to the similarity of their content (this dataset called ‘nsdmeadows’ was released with NSD [32]).

Original: Critically, these images were not seen by any of the response-optimized models.
Revised: Critically, these images were not seen by any of the response-optimized models.

Original: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROI using the correlation distance metric for quantifying pairwise dissimilarities between model-predicted voxel responses.
Revised: We constructed a model Representational Dissimilarity Matrix (RDM) for each visual ROI using the correlation distance metric for quantifying pairwise dissimilarities between model-predicted voxel responses.

Original: We compared model RDMs against human pairwise dissimilarity judgements using the Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.
Revised: We compared model RDMs against human pairwise dissimilarity judgements using the Kendall’s τ coefficient and call this the ‘NSD: perception’ analysis.

Original:  Results
Revised:  Results

Original: Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B,
Revised: Response-optimized models approach the noise ceiling for prediction performance and generalize to novel subjects with low sample complexity We observed a remarkably high noisenormalized prediction accuracy ( ∼ 70-85 % ) using the proposed rotationally-symmetric responseoptimized model in all 5 visual ROIs, including the higher-order ROIs (LO1-2 and VO1-2) (Figure 2B,

Original: Appendix Figure A.2, A.6).
Revised: Appendix Figure A.2, A.6).

Original: Next, we investigated the possible quantitative advantages of learning a rotationally-symmetric feature space against the representations from a standard convolutional architecture with no weight sharing across orientations.
Revised: Next, we investigated the possible quantitative advantages of learning a rotationally-symmetric feature space against the representations from a standard convolutional architecture with no weight sharing across orientations.

Original: Importantly, we observe that the proposed rotationally-symmetric convolutional architecture performs on par with a standard CNN (Figure 2A), even outperforming the latter in certain visual ROIs, despite containing fewer parameters.
Revised: Importantly, we observe that the proposed rotationally-symmetric convolutional architecture performs on par with a standard CNN (Figure 2A), even outperforming the latter in certain visual ROIs, despite containing fewer parameters.

Original: This suggests that sharing weights across filter orientations provides a strong inductive bias, allowing us to fit expressive models efficiently.
Revised: This suggests that sharing weights across filter orientations provides a strong inductive bias, allowing us to fit expressive models efficiently.

Original: We further compared these models against ideal observer and task-optimized models, and found that response-optimized models consistently outperform these baselines in early visual areas.
Revised: We further compared these models against ideal observer and task-optimized models, and found that response-optimized models consistently outperform these baselines in early visual areas.

Original: In high-level visual areas, the response-optimized model significantly outperforms the ideal observer model, while performing on par with or even surpassing some task-optimized models.
Revised: In high-level visual areas, the response-optimized model significantly outperforms the ideal observer model, while performing on par with or even surpassing some task-optimized models.

Original: Next, we assessed how each of these predictive models generalizes to the remaining set of 4 test subjects.
Revised: Next, we assessed how each of these predictive models generalizes to the remaining set of 4 test subjects.

Original: This analysis intended to compare the statistical efficiency afforded by different representations [48].
Revised: This analysis intended to compare the statistical efficiency afforded by different representations [49].

Original: As such, we examined the trend of how much stimulus-response data the models need to achieve a certain level of prediction performance [48].
Revised: As such, we examined the trend of how much stimulus-response data the models need to achieve a certain level of prediction performance [49].

Original: Here, we freeze the shared feature extractor network and vary the amount of stimulus-response pairs from the 4 new subjects to train the linear readouts (Appendix A.7).
Revised: Here, we freeze the shared feature extractor network and vary the amount of stimulus-response pairs from the 4 new subjects to train the linear readouts (Appendix A.7).

Original: We do the same for the task-optimized baselines, where the linear prediction model is fit on new subjects.
Revised: We do the same for the task-optimized baselines, where the linear prediction model is fit on new subjects.

Original: As shown in Figure 2, the responseoptimized models can achieve a significantly high accuracy even when only just 5 % of the training examples from novel subjects ( ∼samples) are used to train the readout.
Revised: As shown in Figure 2, the responseoptimized models can achieve a significantly high accuracy even when only just 5 % of the training examples from novel subjects ( ∼samples) are used to train the readout.

Original: This remarkable generalization of response-optimized networks, exceeding that of task-optimized networks, to novel subjects suggests our models contain appropriate “inductive biases" that improve sample efficiency.
Revised: This remarkable generalization of response-optimized networks, exceeding that of task-optimized networks, to novel subjects suggests our models contain appropriate “inductive biases" that improve sample efficiency.

Original: Computational models generalize to OOD datasets We next investigated out-of-distribution (OOD) test settings to understand the generalization behavior of response-optimized models.
Revised: Computational models generalize to OOD datasets We next investigated out-of-distribution (OOD) test settings to understand the generalization behavior of response-optimized models.

Original: The first OOD setting we consider is the NSD-synthetic experiment, comprising fMRI recordings to synthetic stimuli (samples shown in Figure 3) from the same NSD subjects.
Revised: The first OOD setting we consider is the NSD-synthetic experiment, comprising fMRI recordings to synthetic stimuli (samples shown in Figure 3) from the same NSD subjects.

Original: As shown in Figure 3, response-optimized models significantly outperform task-optimized models (except in region VO) in this OOD domain (‘artificial’ stimuli), suggesting that the mapping from task-optimized DNN features to voxel responses, was domain/dataset-specific.
Revised: As shown in Figure 3, response-optimized models significantly outperform task-optimized models (except in region VO) in this OOD domain (‘artificial’ stimuli), suggesting that the mapping from task-optimized DNN features to voxel responses, was domain/dataset-specific.

Original: Response-optimized models, on the other hand, allow for flexible OOD generalization.
Revised: Response-optimized models, on the other hand, allow for flexible OOD generalization.

Original: Generalization to other novel fMRI datasets, including [49, 50, 51] is shown and discussed in the Appendix B.2.
Revised: Generalization to other novel fMRI datasets, including [50, 51, 52] is shown and discussed in the Appendix B.2.

Original: Learned spatial masks reproduce the fine-grained retinotopic organization of early visual cortex
Revised: Learned spatial masks reproduce the fine-grained retinotopic organization of early visual cortex

Original: The factorized readout in the proposed model allows us to separate spatial and feature selectivity.
Revised: The factorized readout in the proposed model allows us to separate spatial and feature selectivity.

Original: We characterized the spatial tuning of early visual areas by estimating the population receptive field (pRF) from the learned spatial masks of the proposed response-optimized model; we compare the estimated parameters against measurements from the independent NSD pRF localizer experiment that uses artificial stimuli.
Revised: We characterized the spatial tuning of early visual areas by estimating the population receptive field (pRF) from the learned spatial masks of the proposed response-optimized model; we compare the estimated parameters against measurements from the independent NSD pRF localizer experiment that uses artificial stimuli.

Original: The pRF is defined as the region of the visual field within which a stimulus results in increased aggregated activity across a population of neurons, as reflected in fMRI.
Revised: The pRF is defined as the region of the visual field within which a stimulus results in increased aggregated activity across a population of neurons, as reflected in fMRI.

Original: Based on the analysis of these data, each brain voxel’s retinotopic organization can be described with three parameters: the pRF location in polar coordinates (polar angle and eccentricity), and size.
Revised: Based on the analysis of these data, each brain voxel’s retinotopic organization can be described with three parameters: the pRF location in polar coordinates (polar angle and eccentricity), and size.

Original: Using the voxel-wise spatial mask of the readout in our encoding model, we estimated the pRF location as the average of the coordinates weighted by the learned spatial mask value at each grid location.
Revised: Using the voxel-wise spatial mask of the readout in our encoding model, we estimated the pRF location as the average of the coordinates weighted by the learned spatial mask value at each grid location.

Original: To compare the estimated eccentricity against the corresponding measurements from pRF localizers, henceforth called ‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient between the measured and predicted eccentricity arrays in each ROI.
Revised: To compare the estimated eccentricity against the corresponding measurements from pRF localizers, henceforth called ‘eccentricity agreement’ , we calculated the Pearson’s correlation coefficient between the measured and predicted eccentricity arrays in each ROI.

Original: To compare the estimated polar angle values against corresponding measurements from pRF localizer scans, henceforth called ‘polar angle agreement’ , we adopted a measure from circular statistics called circular correlation coefficient [52], due to the circular nature of angular data (Appendix A.4).
Revised: To compare the estimated polar angle values against corresponding measurements from pRF localizer scans, henceforth called ‘polar angle agreement’ , we adopted a measure from circular statistics called circular correlation coefficient [53], due to the circular nature of angular data (Appendix A.4).

Original: It is important to note that the encoding models do not account for the spatial locations of target voxels.
Revised: It is important to note that the encoding models do not account for the spatial locations of target voxels.

Original: Despite this, the estimated spatial topographic organization from the encoding models exhibits remarkable levels of agreement with the fine-scaled retinotopic organization of the cortex without any explicit supervision to do so (Figure 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angle agreement ∈ [0.70-0.82].
Revised: Despite this, the estimated spatial topographic organization from the encoding models exhibits remarkable levels of agreement with the fine-scaled retinotopic organization of the cortex without any explicit supervision to do so (Figure 4A,B): eccentricity agreement ∈ [0.48-0.79] for all 4 subjects whereas the polar angle agreement ∈ [0.70-0.82].

Original: We further compared this agreement on the estimated pRF parameters for the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linear readouts from the fixed convolutional core of the response-optimized model (pre-trained on the initial set of 4 subjects).
Revised: We further compared this agreement on the estimated pRF parameters for the held-out set of 4 NSD subjects by varying the quantity of data used to estimate their linear readouts from the fixed convolutional core of the response-optimized model (pre-trained on the initial set of 4 subjects).

Original: We find that while the agreement increases with the sample size used to fit readouts, even with much fewer samples (e.g. 100), we can still characterize the retinotopic organization (Figure 4C).
Revised: We find that while the agreement increases with the sample size used to fit readouts, even with much fewer samples (e.g. 100), we can still characterize the retinotopic organization (Figure 4C).

Original: This is better than the retinotopic agreement observed for the CORnetS model (which also employed a factorized readout, enabling characterization of spatial tuning (Appendix Figure A.8)).
Revised: This is better than the retinotopic agreement observed for the CORnetS model (which also employed a factorized readout, enabling characterization of spatial tuning (Appendix Figure A.8)).

Original: Previously, retinotopic organization of the human visual cortex has been dominantly studied with spatially modulated stimuli.
Revised: Previously, retinotopic organization of the human visual cortex has been dominantly studied with spatially modulated stimuli.

Original: While encoding models have been shown to be broadly useful for characterizing retinotopy [53, 31], a precise quantification of the agreement between model-estimated and localizer-estimated retinotopic parameters, as well as the effect of sample sizes on this agreement, has been lacking.
Revised: While encoding models have been shown to be broadly useful for characterizing retinotopy [54, 31], a precise quantification of the agreement between model-estimated and localizer-estimated retinotopic parameters, as well as the effect of sample sizes on this agreement, has been lacking.

Original: Here, we present an alternate approach based on naturalistic stimulation and response-optimized encoding models to delineate subject-specific retinotopic maps and precisely quantify the agreement.
Revised: Here, we present an alternate approach based on naturalistic stimulation and response-optimized encoding models to delineate subject-specific retinotopic maps and precisely quantify the agreement.

Original: Remarkably, the retintopic organization estimated from the proposed encoding models is even more spatially smooth than the localizer-based organization, despite not being informed by anatomy.
Revised: Remarkably, the retintopic organization estimated from the proposed encoding models is even more spatially smooth than the localizer-based organization, despite not being informed by anatomy.

Original: Response-optimized models characterize transformations of image representations along the visual cortical hierarchy
Revised: Response-optimized models characterize transformations of image representations along the visual cortical hierarchy

Original: (a) Monotonically increasing alignment with human perception along the ventral visual stream
Revised: (a) Monotonically increasing alignment with human perception along the ventral visual stream

Original: We evaluated models on behavioral data from a Multiple Arrangement task where participants arranged NSD images according to the similarity of their content.
Revised: We evaluated models on behavioral data from a Multiple Arrangement task where participants arranged NSD images according to the similarity of their content.

Original: This behavioral analysis of neural network models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capture human perceptual patterns and image semantics than early visual areas and this quantitative agreement increases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’ analysis discussed in Methods).
Revised: This behavioral analysis of neural network models (Figure 5A) suggests that higher visual areas (LO and VO) more strongly capture human perceptual patterns and image semantics than early visual areas and this quantitative agreement increases monotonically along the ventral visual hierarchy (‘NSD perception’ and ‘NSD semantics’ analysis discussed in Methods).

Original: Importantly, responses patterns in their respective models group semantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region
Revised: Importantly, responses patterns in their respective models group semantically similar categories together, as shown in the t-SNE plot for model-predicted RDM (region

Original: VO).
Revised: VO).

Original: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes group together in the left (Figure 5A), despite wide variations in the visual appearance of these categories.
Revised: For instance, faces are all clustered in the bottom right of the t-SNE plot and scenes group together in the left (Figure 5A), despite wide variations in the visual appearance of these categories.

Original: This is remarkable, given that the models received no supervision from perceptual/semantic measures.
Revised: This is remarkable, given that the models received no supervision from perceptual/semantic measures.

Original: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Appendix Figure
Revised: Human behavioral data (‘nsdmeadows’) appears very similar and is shown in Appendix Figure

Original: A.10.
Revised: A.10.

Original: This suggests that these models, solely optimized for high-level visual cortical response prediction – without any semantic supervision – can even serve as powerful models for uncovering the structure of mental representations.
Revised: This suggests that these models, solely optimized for high-level visual cortical response prediction – without any semantic supervision – can even serve as powerful models for uncovering the structure of mental representations.

Original: Intriguingly, the representational structure of predicted responses exhibits stronger agreement with perceived similarity than the representational structure of measured responses.
Revised: Intriguingly, the representational structure of predicted responses exhibits stronger agreement with perceived similarity than the representational structure of measured responses.

Original: This suggests that both the perceptual and semantic information are likely more explicit in predicted responses (even for held-out data) than measured responses.
Revised: This suggests that both the perceptual and semantic information are likely more explicit in predicted responses (even for held-out data) than measured responses.

Original: One possible explanation is that fMRI measures of brain activity have undesired noise components and predictive models, in particular, response-optimized models as shown here, can denoise this fMRI activity (see also Appendix Figure A.3).
Revised: One possible explanation is that fMRI measures of brain activity have undesired noise components and predictive models, in particular, response-optimized models as shown here, can denoise this fMRI activity (see also Appendix Figure A.3).

Original: (b) Monotonically increasing separability of category information along the ventral visual stream Task
Revised: (b) Monotonically increasing separability of category information along the ventral visual stream Task

Original: -optimized computational models have suggested that the ventral stream hierarchy ‘untangles’ representations through a sequence of processing steps, so that representations of objects and categories that are inseparable at the initial processing stage (e.g., V1) become untangled at the last stage of the hierarchy [54, 23].
Revised: -optimized computational models have suggested that the ventral stream hierarchy ‘untangles’ representations through a sequence of processing steps, so that representations of objects and categories that are inseparable at the initial processing stage (e.g., V1) become untangled at the last stage of the hierarchy [55, 23].

Original: Here, we present a strong evidence for this increased separability through hypothesis-agnostic response-optimized models.
Revised: Here, we present a strong evidence for this increased separability through hypothesis-agnostic response-optimized models.

Original: Previous studies have shown that the representational geometry for the same set of stimuli varies systematically across different cortical areas, providing a useful signature for how information is transformed across different cortical processing streams [54].
Revised: Previous studies have shown that the representational geometry for the same set of stimuli varies systematically across different cortical areas, providing a useful signature for how information is transformed across different cortical processing streams [55].

Original: Here, we employ the representational similarity analysis (RSA) framework to relate emergent representations in response-optimized models of difference brain regions against a ground truth adjacency matrix defined by object category labels.
Revised: Here, we employ the representational similarity analysis (RSA) framework to relate emergent representations in response-optimized models of difference brain regions against a ground truth adjacency matrix defined by object category labels.

Original: Keeping model architecture the same, simply by changing the response targets for an encoding model from voxels in visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extracted representation.
Revised: Keeping model architecture the same, simply by changing the response targets for an encoding model from voxels in visual areas V1 through LO-VO, we observe a drastic change in the geometry of the extracted representation.

Original: The increasingly prominent block-diagonal structure in the RSMs along the ventral visual stream (Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlights that the distributed activity patterns to exemplars of the same category become increasingly more similar; and patterns to exemplars of different categories become progressively dissimilar along the processing stream, strongly supporting the computational goal of the ventral visual stream as a categorization system.
Revised: The increasingly prominent block-diagonal structure in the RSMs along the ventral visual stream (Figure 5C), consistent with categorical distinctions in the THINGS dataset, highlights that the distributed activity patterns to exemplars of the same category become increasingly more similar; and patterns to exemplars of different categories become progressively dissimilar along the processing stream, strongly supporting the computational goal of the ventral visual stream as a categorization system.

Original: We also quantified the separability of categorical information in different visual areas (Figure 5B).
Revised: We also quantified the separability of categorical information in different visual areas (Figure 5B).

Original: Here again, we see that this finding regarding increasing separability of categorical information holds across both datasets containing different visual categories.
Revised: Here again, we see that this finding regarding increasing separability of categorical information holds across both datasets containing different visual categories.

Original: Response-optimized models reveal high-level visual information within ventral-occipital cortex
Revised: Response-optimized models reveal high-level visual information within ventral-occipital cortex

Original: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model and whether they are capable of discriminating between complex categories.
Revised: Next, we wanted to discover what kinds of features emerge in the ventral-occipital region model and whether they are capable of discriminating between complex categories.

Original: Instead of looking at all model neurons emulating voxels, we built a compact neural representational space by passing a large stimulus set ( ∼ 27,000 images from the THINGS database [44]) through the predictive models and then performing Principal Components Analysis (PCA) on the predicted responses of this network (Appendix A.6).
Revised: Instead of looking at all model neurons emulating voxels, we built a compact neural representational space by passing a large stimulus set ( ∼ 27,000 images from the THINGS database [45]) through the predictive models and then performing Principal Components Analysis (PCA) on the predicted responses of this network (Appendix A.6).

Original: Visualizing the images from top concept categories that elicit highest and lowest activation of individual principal components (PCs) reveals complex stimulus dimensions: the first
Revised: Visualizing the images from top concept categories that elicit highest and lowest activation of individual principal components (PCs) reveals complex stimulus dimensions: the first

Original: PC, which explained ∼ 49% of the variance in responses strongly separates faces, animals and bodies from inanimate objects, despite vast variations in the visual appearance of images within sub-categories.
Revised: PC, which explained ∼ 49% of the variance in responses strongly separates faces, animals and bodies from inanimate objects, despite vast variations in the visual appearance of images within sub-categories.

Original: The second PC, accounting for ∼ 16% variance, distinguished objects based on their shape (circular vs. rectilinear), suggesting that such shape features are important for VO.
Revised: The second PC, accounting for ∼ 16% variance, distinguished objects based on their shape (circular vs. rectilinear), suggesting that such shape features are important for VO.

Original: Importantly, these features do not emerge when the same analysis is applied to early visual ROIs (V1-V4).
Revised: Importantly, these features do not emerge when the same analysis is applied to early visual ROIs (V1-V4).

Original: Response-optimized models exhibit a shape bias A striking known discrepancy between human and machine vision is the shape vs. texture bias: human classification decisions are shape-biased whereas CNNs trained for CV tasks tend to classify images by texture [55, 56].
Revised: Response-optimized models exhibit a shape bias A striking known discrepancy between human and machine vision is the shape vs. texture bias: human classification decisions are shape-biased whereas CNNs trained for CV tasks tend to classify images by texture [56, 57].

Original: We study whether the shape bias of humans is reflected in response-optimized models of cortical regions that subserve visual categorization in humans.
Revised: We study whether the shape bias of humans is reflected in response-optimized models of cortical regions that subserve visual categorization in humans.

Original: We first train a linear layer on top of the predicted responses of responseoptimized models to map these responses to visual categories in the
Revised: We first train a linear layer on top of the predicted responses of responseoptimized models to map these responses to visual categories in the

Original: ImageNet-16 database [55] using a small subset of this database.
Revised: ImageNet-16 database [56] using a small subset of this database.

Original: We then assess the transfer performance of response-optimized models for classification on held-out stimuli from the
Revised: We then assess the transfer performance of response-optimized models for classification on held-out stimuli from the

Original: ImageNet-16 dataset as well as the
Revised: ImageNet-16 dataset as well as the

Original: Silhouette dataset described in [55] (Appendix A.10).
Revised: Silhouette dataset described in [56] (Appendix A.10).

Original: We find that the classification performance is significantly better than what would be expected from the same network architecture with a random initialization (Figure 6B), suggesting that the brain response-optimization objective yields effective representations.
Revised: We find that the classification performance is significantly better than what would be expected from the same network architecture with a random initialization (Figure 6B), suggesting that the brain response-optimization objective yields effective representations.

Original: Further, this classification performance is better using models of intermediate and high-level visual areas (V4, LO and VO) than early visual areas, even approaching the transfer performance of an MS-COCO-trained object classification network with the same architecture (Figure 6B,top, see Appendix A.10 for model details), providing further support for the separability of categorical information in high-level, but not early visual cortex [11].
Revised: Further, this classification performance is better using models of intermediate and high-level visual areas (V4, LO and VO) than early visual areas, even approaching the transfer performance of an MS-COCO-trained object classification network with the same architecture (Figure 6B,top, see Appendix A.10 for model details), providing further support for the separability of categorical information in high-level, but not early visual cortex [11].

Original: The LO model has the highest classification performance on the Silhouette dataset (which solely contain shape information), consistent with and providing further evidence for previous hypotheses that LO tracks object shape information [57].
Revised: The LO model has the highest classification performance on the Silhouette dataset (which solely contain shape information), consistent with and providing further evidence for previous hypotheses that LO tracks object shape information [58].

Original: We then evaluate the classification performance of this mapped response-optimized model on the diagnostic texture-shape cue conflict dataset [55] to assess whether the model makes its categorization decisions based on shape or texture (Appendix A.10).
Revised: We then evaluate the classification performance of this mapped response-optimized model on the diagnostic texture-shape cue conflict dataset [56] to assess whether the model makes its categorization decisions based on shape or texture (Appendix A.10).

Original: We follow [55] and define shape bias as the percentage of times the model classifies an image based on its shape over texture, provided it returned either the correct shape or correct texture label for the image.
Revised: We follow [56] and define shape bias as the percentage of times the model classifies an image based on its shape over texture, provided it returned either the correct shape or correct texture label for the image.

Original: A model is shape-biased if its shape bias > 50 % .
Revised: A model is shape-biased if its shape bias > 50 % .

Original: We observed that all response-optimized models exhibit a shape bias (Figure 6B [bottom], Appendix Figure A.1), with this metric ranging from 56.7-69.4 % , suggesting that representations within the ventral visual stream are likely biased towards shapes than textures.
Revised: We observed that all response-optimized models exhibit a shape bias (Figure 6B [bottom], Appendix Figure A.1), with this metric ranging from 56.7-69.4 % , suggesting that representations within the ventral visual stream are likely biased towards shapes than textures.

Original: This is in stark contrast to standard CV models which show a texture bias, for e.g., the same architecture trained for multi-label object classification on MS-COCO shows a strong texture bias and performs significantly worse on the Silhouettes dataset (Figure 6B, gray bar), suggesting that the distribution of NSD stimuli (which are drawn from MS-COCO) is not sufficient to drive the shape bias and the training objective (matching to neural responses rather than object category labels) matters.
Revised: This is in stark contrast to standard CV models which show a texture bias, for e.g., the same architecture trained for multi-label object classification on MS-COCO shows a strong texture bias and performs significantly worse on the Silhouettes dataset (Figure 6B, gray bar), suggesting that the distribution of NSD stimuli (which are drawn from MS-COCO) is not sufficient to drive the shape bias and the training objective (matching to neural responses rather than object category labels) matters.

Original: Further, as shown in Figure 6B (bottom), it is not the linear mapping to neural response but the way features are optimized that explains the differing shape/texture bias of task-optimized and response-optimized encoding models.
Revised: Further, as shown in Figure 6B (bottom), it is not the linear mapping to neural response but the way features are optimized that explains the differing shape/texture bias of task-optimized and response-optimized encoding models.

Original: We next performed a feature visualization analysis to find visual patterns that maximally activate individual model neurons emulating brain voxels in response-optimized encoding models.
Revised: We next performed a feature visualization analysis to find visual patterns that maximally activate individual model neurons emulating brain voxels in response-optimized encoding models.

Original: Following [58], we start with a random noise input x 0 for each model neuron i and iteratively update the input along the gradient ∂a i ∂x to synthesize inputs that would result in higher and higher predicted activation a i for that neuron (Appendix A.9).
Revised: Following [59], we start with a random noise input x 0 for each model neuron i and iteratively update the input along the gradient ∂a i ∂x to synthesize inputs that would result in higher and higher predicted activation a i for that neuron (Appendix A.9).

Original: Visual inspection shows that the features increase in complexity as we go from early to high-level visual ROI models: simple edge-like features or high-low frequency detectors are seen in V1, curves and edges are seen in V2, more complex curvatures in V4 and finally, fully-formed complex shapes like concentric hexagons are seen in LO and VO (Fig.
Revised: Visual inspection shows that the features increase in complexity as we go from early to high-level visual ROI models: simple edge-like features or high-low frequency detectors are seen in V1, curves and edges are seen in V2, more complex curvatures in V4 and finally, fully-formed complex shapes like concentric hexagons are seen in LO and VO (Fig.

Original: 6A, Appendix Fig.
Revised: 6A, Appendix Fig.

Original: A.7).
Revised: A.7).

Original: We also validated this increased complexity using quantitative measures based on compression ratios (Appendix Table A.3).
Revised: We also validated this increased complexity using quantitative measures based on compression ratios (Appendix Table A.3).

Original:  Discussion
Revised:  Discussion

Original: In this study, we leveraged the large sample size and natural variation represented in the Natural
Revised: In this study, we leveraged the large sample size and natural variation represented in the Natural

Original: Scenes Dataset to train deep neural networks directly on the brain response.
Revised: Scenes Dataset to train deep neural networks directly on the brain response.

Original: Concurrently, recent papers [31, 30] have also demonstrated that when trained with large amounts of data such as NSD, the response-optimization approach can perform competitively with state-of-the-art task-optimized models (discussed in more detail in Appendix C.1).
Revised: Concurrently, recent papers [31, 30] have also demonstrated that when trained with large amounts of data such as NSD, the response-optimization approach can perform competitively with state-of-the-art task-optimized models (discussed in more detail in Appendix C.1).

Original: We studied model generalization in various settings: from synthetic stimuli to novel fMRI datasets with different preprocessing.
Revised: We studied model generalization in various settings: from synthetic stimuli to novel fMRI datasets with different preprocessing.

Original: Our results recapitulated key features of early visual areas (e.g. their precise retinotopic organization) and representational properties in higher level areas.
Revised: Our results recapitulated key features of early visual areas (e.g. their precise retinotopic organization) and representational properties in higher level areas.

Original: We also demonstrated that response-optimized models can capture human behavior, including not only object categorization, but also other finegrained measures such as perceived image similarity.
Revised: We also demonstrated that response-optimized models can capture human behavior, including not only object categorization, but also other finegrained measures such as perceived image similarity.

Original: The response-optimization approach, in principle, allows exploring infinite paths in the model space, and when optimized properly, converging towards one that explains maximum possible variance in the data.
Revised: The response-optimization approach, in principle, allows exploring infinite paths in the model space, and when optimized properly, converging towards one that explains maximum possible variance in the data.

Original: Since any features that emerge in these optimized networks are a result of optimization to match neural targets, unconfounded by any other supervision signal, these models can provide stronger tests about the visual representations within different visual areas.
Revised: Since any features that emerge in these optimized networks are a result of optimization to match neural targets, unconfounded by any other supervision signal, these models can provide stronger tests about the visual representations within different visual areas.

Original: In terms of human object categorization behavior, our models lack significantly compared to
Revised: In terms of human object categorization behavior, our models lack significantly compared to

Original: ImageNet-supervised DNN models.
Revised: ImageNet-supervised DNN models.

Original: While part of this may be due to the domain shift, the restricted transfer learning setting (a single linear layer mapping), small size of the transfer set or simply the coarse spatial resolution of fMRI, this gap is worth exploring.
Revised: While part of this may be due to the domain shift, the restricted transfer learning setting (a single linear layer mapping), small size of the transfer set or simply the coarse spatial resolution of fMRI, this gap is worth exploring.

Original: Further, using neural data to build better models for other communities (e.g. computer vision) is another scientifically fascinating direction and can develop reinforcing cycles.
Revised: Further, using neural data to build better models for other communities (e.g. computer vision) is another scientifically fascinating direction and can develop reinforcing cycles.

Original: Studies have shown that explicitly encouraging neural networks to build representations similar V1 representations can improve their robustness under noise and adversarial attacks [59, 60].
Revised: Studies have shown that explicitly encouraging neural networks to build representations similar V1 representations can improve their robustness under noise and adversarial attacks [60, 61].

Original: In a similar vein, regularization with high-level visual cortical activity might help grant models brain-like biases (e.g. shape bias) and close the gap between human and machine vision.
Revised: In a similar vein, regularization with high-level visual cortical activity might help grant models brain-like biases (e.g. shape bias) and close the gap between human and machine vision.

Original: Alternatively, probing the connections in response-optimized models, which already exhibit a shape bias, using model pruning and attribution methods can yield a mechanistic understanding of the shape bias in neural activity and reveal principles that may enable CV models to attain similar representations.
Revised: Alternatively, probing the connections in response-optimized models, which already exhibit a shape bias, using model pruning and attribution methods can yield a mechanistic understanding of the shape bias in neural activity and reveal principles that may enable CV models to attain similar representations.

Original: Recent studies have inspired optimism that studying connection weights to reveal underlying computational principles may be tractable [33, 61].
Revised: Recent studies have inspired optimism that studying connection weights to reveal underlying computational principles may be tractable [33, 62].

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was funded by the following grants: NIH RF1 MH123232, NIH R01 LM012719, NIH
Revised: This work was funded by the following grants: NIH RF1 MH123232, NIH R01 LM012719, NIH

Original: R01 AG053949, NSF CAREER and NSF NeuroNex Grant 1707312 (MS).
Revised: R01 AG053949, NSF CAREER and NSF NeuroNex Grant 1707312 (MS).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] They are described within the Discussion section (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] They are described within the Discussion section (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We don’t foresee any negative societal impacts of this work.
Revised: [N/A] We don’t foresee any negative societal impacts of this work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] I’ve read all guidelines 2.
Revised: [Yes] I’ve read all guidelines 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical results are discussed in this paper.
Revised: No theoretical results are discussed in this paper.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical results are discussed in this paper.
Revised: No theoretical results are discussed in this paper.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Yes, the code is included in the Supplementary Material.
Revised: [Yes] Yes, the code is included in the Supplementary Material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Yes, all training details are included in the Methdos and Supplementary Methods.
Revised: [Yes] Yes, all training details are included in the Methdos and Supplementary Methods.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Yes, they are displayed in all figures.
Revised: [Yes] Yes, they are displayed in all figures.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Yes.
Revised: [Yes] Yes.

Original: All models were trained, validated and analysed on an NVIDIA Titan Xp GPU.
Revised: All models were trained, validated and analysed on an NVIDIA Titan Xp GPU.

Original: Training each response-optimized model on 1 GPU took ∼ 8 hours.
Revised: Training each response-optimized model on 1 GPU took ∼ 8 hours.

Original: Additional details are listed in the Supplementary. 4.
Revised: Additional details are listed in the Supplementary. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] e.g. [47, 62] (b) Did you mention the license of the assets?
Revised: [Yes] e.g. [48, 63] (b) Did you mention the license of the assets?

Original: [No] The assets employed in this study were made publicly available by the creators (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [No] The assets employed in this study were made publicly available by the creators (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [No] The assets employed in this study were made publicly available by the creators (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [No] The assets employed in this study were made publicly available by the creators (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] The assets don’t contain any offensive content to our knowledge.
Revised: [No] The assets don’t contain any offensive content to our knowledge.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]