What can 5.17 billion regression fits tell us about artificial models of the human visual system?
Anonymous Author(s)
AffiliationAddress email
Abstract
Rapid simultaneous advances in machine vision and cognitive neuroimagingpresent an unparalleled opportunity to assess the current state of artificial modelsof the human visual system.
Here, we perform a large-scale benchmarking analysisof 72 modern deep neural network models to characterize with robust statisticalpower how differences in architecture and training task contribute to the predictionof human fMRI activity across 16 distinct regions of the human visual system.
Wefind: one, that even stark architectural differences (e.g. the absence of convolutionin transformers and MLP-mixers) have very little consequence in emergent fits tobrain data; two, that differences in task have clear effects–with categorization andself-supervised models showing relatively stronger brain predictivity across theboard; three, that feature reweighting leads to substantial improvements in brainpredictivity, without overfitting – yielding model-to-brain regression weights thatgeneralize at the same level of predictivity to brain responses over 1000s of newimages.
Broadly, this work presents a lay-of-the-land for the emergent correspondences between the feature spaces of modern deep neural network models and therepresentational structure inherent to the human visual system.
Introduction
The pace of progress in computer vision poses a practical challenge for neuroscientists seeking toassess state-of-the-art models in their ability to explain visual representation and behavior.
New highperforming models are released on a near-daily basis and recent innovations (e g. in self-supervisedlearning [1, 2]) have created myriad new opportunities for productive synergy between the fields ofbiological and machine vision.
As such, methods for comparing the brain-predictivity of artificialmodels using a predefined analysis pipeline (“neural benchmarking”) are critical in helping discernthe algorithmic innovations that may be meaningful with respect to the study of brain function.
Existing public neural benchmarking datasets have been limited to mouse and primate neurophysiology (3, 4).
Recent advances in the scale and quality of human neuroimaging datasets (5–7) nowpresent an opportunity to rigorously assess the state of deep neural network modeling as applied tothe human visual system.
Here we present a large-scale benchmark of dozens of state-of-the-art deep neural network modelsin their prediction of human brain activity across the visual hierarchy.
Aiming for coverage, oursurvey attempts to document the current trends in how well different kinds of models, varying in bothtask and architecture, learn features with brain-like response signatures.
Our results complementprior work examining different model predictivities (8–10), but at a significantly larger scale, and incorporating a set of more modern models not yet fully accounted for in the benchmarking literature(e.g. self-supervised models and vision transformers).
Methods
As the target of our neural benchmark, we use the Natural Scenes Dataset (NSD, [5]), a recent fMRIdataset representing the most extensive sampling of visual responses in individual participants todate (30,000 stimuli viewed per subject; 73,000 unique images total).
Here we analyze only a smallfraction of this dataset, focusing on responses to 1,000 COCO stimuli that were shown to 4 subjectsat least 3 times, in a subset of ROIs along the visual hierarchy.
We compare these responses with theresponses of 72 modern DNNs that vary in task and architecture (see Appendix for details).
We employ two methods for mapping the activations of model features within a layer to regionsof the brain – classical representational similarity analysis (RSA, [11]) and voxelwise-encoding(re-weighted)
RSA (12).
Classical RSA considers all of the features from a given model layer equallyin computing the image-wise representational dissimilarity matrix (RDM), which is directly comparedwith a given neural RDM.
This method requires a fully-emergent match in population-level geometrybetween a neural ROI and the full set of units in a model layer.
Voxelwise encoding RSA (veRSA), on the other hand, takes advantage of feature reweighting toidentify different model subspaces that correspond to the variance in different brain regions (13).
Toimplement voxel-wise encoding RSA, we use an efficient high-throughput model-fitting procedure,first applying leave-one-out cross-validated ridge regression to map between a given model featurespace and the observed univariate activity pattern of each voxel; once we’ve collected a set ofpredictions for the patterns of activity for each voxel in a given ROI, we compute an RDM fromthese predictions and compare that RDM to the RDM in the brain.
This re-weighted RSA procedurerequires massive parallelization, and entails performing a total of around 5.17 billion regressionfits (calculated by multiplying the total number of model layers we analyze by the total number ofvoxels under consideration from the brain dataset).
To assuage concerns of overfitting, we validatethe robustness of our fitted regressions by testing their generalizability to 1000 independent imagesentirely removed from the training procedure.
Results3.1 Hierarchical Correspondence
As a first step and sanity check, we ask: Does the seminal finding that the information processinghierarchies in deep nets recapitulate the information processing hierarchy in the human visual system(14–16) hold at scale and across a significantly diverse population of models?
The answer is aresounding affirmative ( Figure 1 ): Using a purely data-driven aggregation procedure, we show thatthe relative depth of the best-fitting model layer for each ROI seems to re-capitulate the humanvisual hierarchy (e.g. early visual areas, followed by category-selective regions).
This hierarchicalconvergence holds even when breaking down the models by broad, divergent classes of architecture.
3.2 Architecture Variation
How do models with different architectures compare in their ability to predict the structure of humanbrain responses across the visual system?
Our particular survey of models, chosen deliberatelyto reflect the diversity of modern object recognition (ImageNet-trained) architectures, allows fornumerous subdivisions, but perhaps the most prominent is between convolutional architectures (e.g.
VGG, ResNet, MobileNet, n = 24 ), vision transformers (e.g. Visformer, DeIT, n = 13 ) and MLPmixers (e.g. ResMLP, gMixer, n = 5 ).
The latter two of these are more recent advents of computervision, and are defined by the lack of a convolutional inductive bias – once considered a cornerstoneof the link between biological and machine vision.
Comparisons between these architectures (acrossboth classical RSA and voxel-encoding RSA) are shown in Figure 2.
To test for differences in predictivity, we use nonparametric ANOVAs.
Without reweighting (classical RSA), there is a significant difference across ConvNets, MLP-Mixers, and Transformers( χ 2 Kruskal-Walils (2) = 10 .
14 , p Holm = 0 .
02 , (cid:98) ε 2 ordinal = 0 .
27 , CI 95% [0 . 09 , 0 . 49] ) in early visual areas, driven by a significant pairwise advantage of ConvNets over Transformers.
With reweighting (veRSA), this difference disappears.
Without reweighting, there is no significant differencebetween architectures in higher-level cortical areas.
With reweighting, there is a difference( χ 2 Kruskal-Wallis (2) = 10 .
59 , p Holm = 0 .
02 , (cid:98) ε 2 ordinal = 0 .
26 , CI 95% [0 . 09 , 0 .
56] ), driven this time bythe pairwise superiority of both ConvNets and MLP-Mixers over Transformers.
Behind these apparently significant effects is the numerical reality that the raw effect sizes inboth cases is effectively negligible – less than r Pearson = 0 .
01 and r Pearson = 0 .
02 , respectively.
As such, the most striking effect here is not that of architecture, but of mapping method, whichsubstantially augments the predictive power of every model in our survey (with average gains ofr Pearson = 0 .
160 , CI 95% [0 .
166] across model and ROI).
In the most notable case, modelsin EBA experience average gains of r Pearson = 0 . 265 .
These improvements dwarf any differenceattributable to architecture, and underscore an important point: despite dramatic differences in thedesign and algorithmic inductive biases of ConvNets, MLP-Mixers, and Transformers, there is littleconsequence on the resulting brain predictivity (regardless of mapping method).
3.3 Task Variation
How does brain predictivity vary as a function of task?
For a window into this question we considerthe 24 models from the Taskonomy project (17).
These models share the same base architecture(ResNet-50) and visual diet, but are trained on 1 of 24 popular computer vision tasks.
These tasksare organized into 4 different categories (2D, 3D, Semantic and Geometric) according to what theauthors of the Taskonomy project call the models’ ’transfer affinity’ – the degree to which a modeltrained on one task supports transfer learning to another.
The prediction levels of these models forboth classical and voxel-wise encoding RSA are shown in Figure 3.
Without reweighting, there is considerable variability across ROI in the tasks that are most predictiveof the brain, but the differences between the best task and the second-best task is minimal in most cases.
In V2, for example, a 2D task (edge detection) is the most predictive of the tasks at r Pearson = 0 .
226 ,but is closely followed by a 3D task (Keypoints) at r Pearson = 0 . 220 .
As in the case of architecture, feature reweighting (veRSA) leads to uniform improvement acrossmodels.
Strikingly, however, object and scene classification gain disproportionately.
The gains forobject recognition are so substantial that it becomes the single most predictive task for all brain areas,often dominating by an impressively large margin, with a mean gain over the next best task (apartfrom scene classification) of r Pearson = 0 .
127 , CI 95% [0 . 122 , 0 .
131] ) across all ROIs.
While these results point strongly to an advantage of category supervision in the formation of neurallypredictive representation (at least in the case of veRSA), the self-supervised models (absent from
Taskonomy) in our survey allow us to delve more deeply into whether the classification objective isthe key driver of neural predictivity, or whether category-supervised models derive their advantagefrom the set of invariances that they learn in service of classification.
The predictive power of our self-supervised models strongly suggest the latter: regardless of mappingmethod, self-supervised models (especially recent contrastive ResNet-50 models such as SimCLRand BarlowTwins) tend to show a small but statistically significant advantage over a (recentlyrevamped) category-supervised ResNet-50 [18].
For example, averaging across brain ROIs, SimCLReeks out a mean gain of r Pearson = 0 .
013 , CI 95% [0 . 0106 , 0 . 0192]
in weighted RSA and a gain ofr Pearson = 0 .
006 , CI 95% [0 . 002 , 0 . 008] in classical RSA.
While these results should not be interpretedas indicating superiority of self-supervision over category-supervision, they do indicate parity inprediction levels – a win for ethological plausibility (12, 19).
3.4 Generalization Tests
The sheer quantity of regression fits required to summarize the predictive performance of our modelset, and the vast number of dimensions relative to data points, may raise concern: is this deepencoding pipeline massively overfitting, in spite of our cross-validation procedures?
Or, are theestimates we derive truly a reasonable approximation (given the linking assumptions inherent in theanalysis) of a given model’s brain predictivity?
To address this concern, we conducted a separate generalization test for each model, in which weselected the best performing layer (according to the original LOOCV score from our regressionprocedure) per subject, per ROI.
For these layers, we then use reweighted RSA to compare brain andmodel feature spaces using a set of 1000 entirely held-out test images per subject.
These images werenever referenced or incorporated during training, and prediction scores on these images thus providea measure of "pure" generalization.
Even with this more stringent test, we found little-to-no drop in accuracy in predicting brain representation evoked by the 1000 unique test images per fMRI subject.
When aggregating across subjects,models, and ROI, for example, the mean decrease in score on the unseen images was less than 1%( r Pearson = 0 .
0095 , CI 95% [0 . 00422 , 0 . 0153] ).
By adding a mere 103 million regression fits to ourinitial total of 5.17 billion, then, we can thus confirm definitively that our encoding models generalizeto previously-unseen data.
(A more detailed figure showing generalization across specific subjectsand ROIs is shown in the Appendix.)
Discussion
So what can we learn about the human visual system from 5.17 billion regression fits?
Broadly, itseems, there are two sets of answers, one more pessimistic, one more optimistic.
On the side ofpessimism, the lack of variation across architecture suggests that massive innovations in computervision may often yield little to no change in our ability to predict the representational structure ofbiological vision, disrupting what was once prophesied to become a glorious feedback loop between neuroscientific insight and engineering innovation.
What’s more, the frequent variability in interpretation across mapping method seems a potential pitfall if not accounted for with greater vigilanceand attention to theoretical commitment.
On the side of optimism, it appears that more general,algorithmic correspondences between DNNs and brains (especially in terms of the informationprocessing hierarchy) persist in spite of an increasingly rapid shift away from biological plausibilityin engineering.
In opposite direction of this shift is a promising move towards ethological plausibility– many cutting-edge models no longer rely on learning targets humans almost certainly do not share(e.g.
full category supervision).
Not coincidentally, these models appear to be competitive predictorsof brain activity.
Current models are still far from capturing the kaleidoscopic complexity of biological visual systems.
Our goal in pursuing this large-scale benchmark is not to discern the "best" model of vision, butrather to clarify what kinds of things are and are not important for building next-generation perceptualmodels that will push our understanding of human vision further.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See discussion section.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] The code and the data will be released publicly upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Methods section and the Appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See figures, figure captions, Methods section and theAppendix.(d)
Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix section describing compute required
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All details relating to the consent processes for human subjects
included in the Natural Scenes Dataset can be found in
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Appendix
A.1 Human Brain Data
The Natural Scenes Dataset (NSD) is the largest effort to date to measure human brain responses withfunctional magneic resonance imaging (fMRI), reflecting measurements of 73,000 unique stimulifrom the Microsoft Common Objects in Context (COCO) dataset ([20]) at high resolution (7T fieldstrength, 1 .
33 s TR, 1 . 8 mm 3 voxel size).
In the present work, we analyze only a small fractionof this dataset, focusing on responses to images that enable direct comparison between data fromdifferent subjects.
That is, we focus on the 1000 COCO stimuli that overlapped between subjects (the"shared1000" images), and limit analyses to the 4 subjects (subjs 01, 02, 05, 07) for whom all 3 imagerepetitions are available for the shared1000.
For the generalization tests, we also select a randomunique set of 1000 images for each subject; these were not included in the "shared1000."
All responseswere estimated using a custom GLM toolbox ("GLMsingle" [21]), which was applied during thepreprocessing of NSD time-series data, featuring optimized denoising and regularization procedures,to accurately measure changes in neural activity in response to each experimental stimulus.
We focus our analyses to voxels within a set of predefined functional ROIs that span the visualhierarchy (see [5] for details on the procedures used to define the ROIs).
Further, to maximize
SNR of the target data, we implement a reliability-based voxel selection procedure [22] that isolatesregions of the brain containing stable structure in their responses.
To compute the split-half reliability a given voxel, we use 1,000 images from each subject (independent from the shared1000,and from all images included in our main analyses and generalization tests), and take the average correlation in univariate response profiles over each pair of available image repetitions (e.g.mean ( r ( rep 1 , rep 2) , r ( rep 2 , rep 3) , r ( rep 1 , rep 3) ).
ROI voxels exceeding a reliability threshold of
Pearson r = 0 .
1 were included in subsequent analyses.
These procedures yield a matrix of dimension(images, voxels, repetitions) for each subject’s ROI, and we average over the 3 repetitions to yield thefinal ROI data input into our neural benchmarking pipeline.
A.2 Candidate Deep Neural Network Models
In total, we survey a set of 72 distinct models (110 including the randomly-initialized versions ofcertain of these models).
These models are sourced from four different repositories: the Torchvision(PyTorch) model zoo [23]; the pytorch-image-models (timm) library [24]; the VISSL (self-supervised)model zoo [25]; and the Taskonomy (visualpriors) project [17, 26, 27].
The first two of theserepositories offer pretrained versions of a large number of object recognition models with varyingarchitectures: including (classic and modern) convolutional networks, vision transformers, and MLPmixers.
For each of these ’ImageNet’ (object recognition) models, we include one trained and onerandomly initialized variant (using the initialization scheme the model authors recommended) so as toassess the impact of ImageNet training on brain prediction, and as a sanity check.
The self-supervisedmodels are mainly variants on a popular convolutional architecture (ResNet-50), though do includesome transformers (the ’DINO’ models).
The Taskonomy models consist of a core encoder-decoderarchitecture trained on 24 different common computer vision tasks, ranging from autoencoding toedge detection.
These models are engineered in such a way that only the architecture of the decoder varies across task, allowing us to assess (after detaching the encoder)
what effect different kinds oftraining has on predictive power, independent of model design.
A.3 Benchmarking Pipeline
Feature Extraction For each of our deep neural network models, we extract features in response toeach of our probe stimuli at each distinct layer of the network.
Importantly, we define a layer hereas a distinct computational (sub)operation.
This means, for example, that we treat convolution andthe rectified nonlinearity that follows it as two distinct feature maps.
This is especially relevant inthe case of transformers, where the features inherent to the key - query - value computation of theattention maps often differ substantially.
At the end of our feature extraction procedure, we have foreach model and each model layer, a matrix of features of the dimensions number of images x numberof flattened features from a given layer .
Classical RSA (cRSA)
As a first method of mapping deep neural network responses to voxelresponses, we use classical RSA, a nonparametric mapping method that quantifies the emergentsimilarity of the ’representational geometry’ between two feature spaces, regardless of origin.
Tocompute this metric, we construct representational dissimilarity matrices (RDMs) using the pairwisecorrelation distance (1 - Pearson’s r ) between the responses of a given neural ROI (image by voxel) ora given model layer (image by unit) for all images being considered.
We then compare these RDMsby taking a second-order correlation (Pearson’s r ) between the flattened upper-triangular portionof each.
This ultimately yields a matrix of correlation scores of dimension: (number of subjects xnumber of ROIs x number of model layers x number of models).
Classical RSA reflects the extent towhich the representational structure in each model layer naturally recapitulates the representationalstructure in a visual cortical ROI, without alteration or feature reweighting.
Voxelwise Encoding RSA
The following procedure yields the billions of regression fits we referencein the title.
The pipeline works as follows: first, we fit a regression for each voxel as a weightedcombination of model layer features.
Given that the number of features in a layer sometimes numberin the millions, we employ sparse random projection [28] as a dimensionality-reduction procedure,and then use ridge regression as a linear model to relate the model feature space to each voxel’s tuningfunction.
Then, we use the voxel-encoding models to generate predicted activation profiles to thecomplete set of held-out images, and correlate the subsequent predicted representational similaritystructure to that of the brain.
For additional detail, see Section A.4.
We emphasize that this method contrasts with popular practices in primate and mouse benchmarking,which treat predictivity of unit-level univariate response profiles as the key measure.
However, fMRIaffords more systematic spatial sampling over the cortex.
Thus, for the present analysis, ratherthan taking the aggregate of single voxel fits as our key measure, we choose to treat the populationrepresentational geometry over each ROI as our critical target for prediction.
This multi-voxelsimilarity structure provides different kinds of information about the format of population-levelcoding than do individual units.
Noise Ceilings and Reference Metrics
While powerful in the quantity and diversity of its images, the number of repetitions in imagepresentation (3 per image) in the NSD dataset leaves little room to estimate a noise ceiling per voxelwith standard split-half reliability methods.
Thus, as a reference metric for how well our modelsare doing overall, we use inter-subject predictivity: a measure of how well the brain of one humanpredicts the brain of another.
Here, we took the average pairwise correlation of the individual subject
RDMs in a given ROI.
For a more in-depth discussion of ceilings and reference points, as well asexperimental alternatives, see Section A.5.
A.4 Voxelwise Encoding RSA In-Depth
To predict the activity profile of each voxel, we first use Sparse Random Projection (SRP) [28] toproject the model features generated in response to our 1000 probe images into a lower-dimensional space.
We use a dimensionality of 5960 projections–a number we chose a priori using the JohnsonLindenstrauss lemma, which mathematically guarantees the preservation of pairwise distances in agiven space of operations (with a minimal distortion defined by a hyperparameter epsilon, which weleave in all cases at the scikit-learn default of 0.1).
We then perform a leave-one-out cross-validated(LOOCV) ridge regression (cross-validating over images) to map these projections to the responsesof our voxels, obtaining a vector of predicted voxel responses that we then correlate with the truevoxel responses to obtain a score per voxel per model layer.
This leave-one-out cross-validation is performed in a single matrix operation often referred to asgeneralized cross-validation, and is numerically equivalent to iterative leave-one-out, but is effectivelyinstantaneous.
We iterate this regression procedure until we have a score for all voxels and all modellayers.
No hyperparameter selection was performed over the course of the benchmarking, apart froma minimal, exploratory grid search for a lambda parameter (of values: 1 e 1 , 1 e 2 , 1 e 3 , 1 e 4 , 1 e 5 , 1 e 6 , 1 e 7 )on an AlexNet model (which we subsequently excluded from the main analysis).
Thus, all featurespaces were projected to 5960 sparse random projections, and all regressions were run with a lambdapenalty of 1 e 5 .
Rather than taking single-voxel fits as our key measure, we consider the geometry of the populationacross the larger region of interest as a critical target for prediction.
To do so, we use the predictedresponses from our voxel-wise encoding method to generate predicted representational dissimilaritymatrices.
The logic behind this procedure is effectively to dispense with or otherwise transformirrelevant features from the network via reweighting, such that new images are cast into a weightedsubspace of the original feature space.
The representational geometry of this subspace serves as thecomparison to the brain.
At the end of this procedure, we obtain a matrix of correlation scores ofdimension (number of subjects x number of ROIs x number of model layers x number of models).
A.5 Intersubject Predictivity and the Noise Ceiling
In general, the purpose of a noise ceiling is to estimate (at the level of an individual unit of prediction)how reliable the response in that unit is across time.
This metric allows us to then quantify how wellour response data at one point predicts our response data at another.
One example such measurerelevant to fMRI is the Spearman-Brown-corrected split-half reliability of a voxel response oversequential presentations of the same stimuli.
However, this method tends to underestimate true voxelreliability in regimes with few presentations.
The alternative we have provided here – the pairwise inter-subject representational similarity reference – is straightforward in its calculation, and computationally equivalent to the procedure forbenchmarking the models with classical RSA (which is to say, that subject RDMs and model RDMswere computed in the exact same way, and compared using the same correlation metric).
As a reference point for weighted RSA, however, this threshold is perhaps a bit misleading – sinceonly the models benefit from the reweighting.
One possible alternative, similar to work done recentlyin the neural network modeling of mouse visual cortex [30], is to directly incorporate the neuralactivity of human brains into a regression procedure wherein the regressand is the neural activity of atarget subject and the regressors are the neural activities of other subjects.
This procedure has theadvantage of equating the set of computational (sub)operations that map model feature spaces to thebrain, and of providing similarly intuitive targets that undergird inferences over how much of thevariance in a target biological system we can capture with a system that is decidedly not biological.
As a preliminary test of reweighted inter-subject predictivity, we consider another version of thepairwise metric above, predicting single subjects using data from other single subjects.
For each pairof subjects, in which one is the target and the other is the contrast, we iterate over ROIs, gatheringall voxels from the contrast’s ROI to serve as regressors in the prediction of activity in each of the target’s ROI voxels.
We repeat this procedure until we have predicted all voxels in a given targetsubject with all possible contrast subjects.
The mapping procedure in this case is exactly the same asit was for the mapping of models, controlled even to the hyperparemeter: we project the ROI voxelactivity from the contrast subject to 5960 sparse random projections, and regress these subjects to thetarget voxel with a ridge regression set to a lambda penalty of 1 e 5 .
While this method equates each computational (sub)operation between model and human, theintersubject predictivity threshold it establishes is even lower than the version without reweighting.
One reason for this may be that the brain activity from a single subject does not provide a sufficientbreadth of variance to benefit from the reweighting.
As a first pass at rectifying this issue, we deviseda new measure, predicting each voxel from the concatenated activity of all voxels from all othersubjects in the target ROI, effectively creating a multi-human reference.
While we are continuingto assess, conceptually, whether such a reference point may be useful, the estimates it producesfor individual subjects are indeed far higher than the estimates of either the unweighted individualsubject-to-subject comparison or the corresponding weighted comparison, and is in most cases farhigher than the observed levels of model prediction.
Figure 5 shows a comparison between thedifferent kinds of human reference points we compute.
A.6 Generalization Scores across Subject + ROI
Figure 4 shows the generalization scores across individual subjects and individual ROIs.
A.7 Compute Required
We used a single machine with 8 Nvidia RTX 3090 GPUs, 755gb of RAM, and 96 CPUs.
GPUs wereused only for extracting model activations, and could (without major slowdown) be removed fromthe analytic pipeline.
Dimensionality reduction and regression computations were CPU and RAMintensive.
Replicating