Human alignment of neural network representations
Abstract
Today’s computer vision models achieve human or near-human level performance across a wide variety of vision tasks.
However, their architectures, data, and learning algorithms differ in numerous ways from those that give rise to human vision.
In this paper, we investigate the factors that affect alignment between the representations learned by neural networks and human concept representations.
Human representations are inferred from behavioral responses in an odd-one-out triplet task, where humans were presented with three images and had to select the oddone-out.
We find that model scale and architecture have essentially no effect on alignment with human behavioral responses, whereas the training dataset and objective function have a much larger impact.
Using a sparse Bayesian model of human conceptual representations, we partition triplets by the concept that distinguishes the two similar images from the odd-one-out, finding that some concepts such as food and animals are well-represented in neural network representations whereas others such as royal or sports-related objects are not.
Overall, although models trained on larger, more diverse datasets achieve better alignment with humans than models trained on ImageNet alone, our results indicate that scaling alone is unlikely to be sufficient to train neural networks with conceptual representations that match those used by humans.
Introduction
Representation learning is a fundamental part of modern computer vision systems, but the paradigm has its roots in cognitive science.
When Rumelhart et al.
[56] developed backpropagation, their goal was to find a method that could learn representations of concepts that are distributed across neurons, similarly to the human brain.
The discovery that representations learned by backpropagation could replicate nontrivial aspects of human concept learning was a key factor in its rise to popularity in the late 1980s [64, 45].
A string of empirical successes has since shifted the primary focus of representation learning research away from its similarities to human cognition and toward practical applications.
This shift has been fruitful.
By some metrics, the best computer vision models now outperform the best individual humans on benchmarks such as ImageNet [59, 7, 68].
However, the extent to which the conceptual representations learned by these high-performing vision models align with those used by humans remains unclear.
Do models that are better at classifying images naturally learn more human-like conceptual representations?
Prior work has investigated this question indirectly, by measuring models’ error consistency with humans [17, 52, 20] and the ability of their representations to predict neural activity in primate brains [70, 23, 58], with mixed results.
Here, we approach the question of alignment between human and machine representation spaces more directly, using human similarity judgments collected from an odd-one-out task, where humans saw triplets of images and selected the image most different from the other two [28].
These similarity judgments allow us to infer that the two images that were not selected are closer to each other in an individual’s concept space than either is to the odd-one-out.
We define the odd-one-out in the neural network representation space analogously, and measure neural networks’ alignment with human similarity judgments in terms of their odd-one-out accuracy , i.e., the accuracy of their odd-one-out “judgments” with respect to humans’, under a wide variety of settings.
Based on these odd-one-out accuracies, we draw the following conclusions:
• Scaling ImageNet models improves ImageNet accuracy, but does not consistently improve alignment of their representations with human similarity judgments.
Differences in alignment across ImageNet models appear to arise primarily from differences in objective functions rather than from differences in architecture or width/depth.
• Models trained on image/text data, or on larger, more diverse classification datasets than ImageNet, achieve substantially better alignment with humans.
• We use a sparse Bayesian model of human mental representations [44] to partition triplets by the concept that distinguishes the odd-one-out.
While food and animal-related concepts can easily be recovered from neural net representations, human alignment is weak for dimensions that depict sports-related or royal objects, especially for ImageNet models.
We discuss related work more thoroughly in Appendix A.
Methods
[Figure]
Data
The images used in this paper are taken from the THINGS database [27].
THINGS consists of a collection of 1,854 object categories, i.e., concrete nameable nouns in the English language, along with representative images for these categories.
THINGS was curated to include categories that can be easily identified as a central object in a natural image.
Hebart et al.
[28] collected similarity judgments from human participants on categories in THINGS , which they then used to derive concept representations.
These similarity judgments came in the form of responses to a triplet task .
In a triplet task, images from three distinct categories are presented to a participant, from which the participant selects the image that is most different from the other two (or equivalently the pair of images that are most similar).
The authors collected 1.46 million unique responses crowdsourced from 5,301 workers.
See Figure 1 for an example triplet.
For presentation purposes, we have replaced the images used in Hebart et al.
[28] with images similar in appearance that are licensed under CC0 [63].
Models In our evaluation, we consider a diverse set of pretrained neural networks, including a wide variety of self-supervised and supervised models trained on ImageNet-1K and ImageNet-21K [12]; a Vision Transformer trained on the proprietary JFT-3B dataset [72]; and models that were trained on both image and text data such as CLIP [51], ALIGN [32], and BASIC [49].
See Table C.1 for a comprehensive list of all models.
In our plots, we determine the ImageNet top-1 accuracy for networks not trained on ImageNet-1K by training a linear classifier on the network’s penultimate layer using L-BFGS [42].
Zero-shot odd-one-out accuracy We examine the extent to which the odd-one-out can be identified directly from the similarities between images in models’ representation spaces.
Given representations x 1 , x 2 , and x 3 of the three images that make up the triplet, we first construct a similarity matrix
S ∈ R 3 ×
where S i,j := x Ti x j / ( ∥ x i ∥
∥ x j ∥ 2 ) , the cosine similarity between a pair of representations.
We identify the closest pair of images in the triplet as arg max i,j>i S i,j ; the remaining image is the odd-one-out.
We define zero-shot odd-one-out accuracy as the proportion of triplets where the odd-one-out identified in this fashion matches the human odd-one-out response.
When evaluating zero-shot odd-one-out accuracy, we report the better of the accuracies obtained from representations of the penultimate embedding layer and logits (if the network has a logits layer).
As we show in
Figure C.1, representations obtained from earlier network layers perform worse.
Probing In cases where a model’s zero-shot accuracy is low, decoding the information necessary for downstream tasks may only require a linear transformation.
Generally, linear probing yields insights into the information encoded in neural net’s representation [60, 2].
To perform linear probing, we formulate the notion of the odd-one-out probabilistically, as in Hebart et al.
Given similarity matrix S and a triplet { i, j, k } (here the images are indexed by natural numbers), the likelihood of a particular pair, { a, b } ⊂ { i, j, k } , being most similar, and thus the remaining image being the odd-one-out, is modeled by the softmax of the image similarities,
[Equation]
We learn the linear transformation that maximizes the log-likelihood of the triplet odd-one-out judgments plus an ℓ 2 regularization term.
Specifically, given triplet responses ( { a s , b s } , { i s , j s , k s } ) ns =
we find a square matrix W yielding a similarity matrix S ij =
( W x i ) T ( W x j ) that optimizes
[Equation]
Here, we determine λ via grid-search during k -fold cross-validation (CV).
To obtain a minimally biased estimate of the odd-one-out accuracy of a linear probe, we partition the m objects into two disjoint sets.
Experimental details about the optimization process, k -fold CV, and how we partition the objects can be found in Appendix B.1 and in Algorithm 1 respectively [Figure]
VICE Several of our analyses make use of human concept representations obtained by Variational
Interpretable Concept Embeddings (VICE), an approximate Bayesian method for finding concept representations from human odd-one-out responses in a triplet task [44].
VICE uses mean-field VI to yield a sparse representation for each image that best explains these responses.
VICE does not consider the content of the images and cannot provide representations for novel images.
VICE shows high reproducibility of representations across different random initializations, and has strong predictive power, achieving an odd-one-out accuracy of ∼ 64% on THINGS , which is only marginally lower than the estimated ceiling accuracy of 67 .
Experiments 3.1 Odd-one-out vs. ImageNet accuracy
[Figure]
ImageNet accuracy generally is a good predictor for transfer learning performance [35, 13, 16].
Thus, we evaluate zero-shot odd-one-out accuracy for all models in Table C.1 and compare it with their ImageNet top-1 accuracy.
Figure 2 shows results for both the THINGS triplet task (left) as well as a triplet task constructed using thecoarse classes of the CIFAR100 dataset (right).
To generate CIFAR -100 triplets, we select two images from the same coarse class and one odd-one-out image from a different class; see Appendix D for further details.
While ImageNet accuracy is highly correlated with odd-one-out accuracy for the CIFAR -100 coarse task ( r = 0 . 802 ), its correlation with accuracy on human odd-one-out judgments is much weaker ( r = 0 . 114 ).
This raises the question whether there are model, task, or data characteristics that influence human alignment.
Architecture or objective?
The top row of Figure 3 shows odd-one-out accuracy as a function of
ImageNet performance for models from two recent studies that investigated the transferability of
ImageNet pretrained representations that vary only in the architecture or training objective, with all [Figure] 21K) other hyperparameters fixed [35, 36].
We find that models with the same architecture (ResNet-50) trained with different data augmentation or objective functions [36] yield substantially different zeroshot odd-one-out accuracies.
Conversely, models with different architectures trained with the same objective function [35] - softmax cross-entropy -, achieve similar odd-one-out accuracies, although their ImageNet accuracies vary significantly.
This suggests that architecture does not affect oddone-out accuracy, while the training objective function and the augmentation strategy appear to have a significant effect.
Self-supervised learning The plot in the bottom left corner of Figure 3 compares zero-shot oddone-out accuracy of different SSL models with their linear probing ImageNet performance.
The non-Siamese models Jigsaw [46] and RotNet [21] show substantially worse alignment with human judgments than other SSL models.
This is not surprising given their poor performance on ImageNet.
For the Siamese methods SimCLR [10], MoCoV2 [26], Barlow Twins [71], SwAV [9], and VICReg [5], however, ImageNet performance does not correspond to alignment with human judgments.
Model capacity The graph in the bottom right corner of Figure 3 plots zero-shot odd-one-out accuracy against the number of model parameters for a subset of ImageNet models.
While one typically observes a positive correlation between model capacity and task performance in computer vision, we do not observe any relationship between model width/depth and odd-one-out accuracy.
3.2 How much alignment can a linear probe recover?
[Figure]
Probing and zero-shot odd-one-out accuracies are positively correlated, in the embedding (Figure 4; r = 0 . and logits layer (Figure E.2; r = 0 . 963 ).
However, there are models in Figure 4 that show poor zero-shot and strong linear probing odd-one-out accuracies, such as
ALIGN, SWAV, EfficientNet B4 and ViT-L/16.
ALIGN is probably the most interesting candidate.
Although its zero-shot odd-one-out accuracy is average, this image/text model achieves the highest probing odd-one-out accuracy across all evaluated models.
As we show in Appendix E, the relationship between probing odd-one-out accuracy and ImageNet accuracy is similar to the relationship between zero-shot odd-one-out accuracy and ImageNet accuracy described above.
The correlation between ImageNet accuracy and probing odd-one-out accuracy is still weak ( r = 0 . 213 ).
Probing reduces the variance in odd-one-out accuracy among networks trained with different loss functions and Siamese self-supervised learning methods, but there is still no clear improvement in odd-one-out accuracy with better-performing architectures or larger model capacities.
In the following analysis, we evaluate both zero-shot and linear probing odd-one-out accuracy for individual human concepts.
We partitioned the original triplet dataset according to the VICE dimension shared between the two more similar images; see Appendix F for details.
In Figure 5, we show zero-shot and linear probing odd-one-out accuracies for three VICE dimensions, for all models listed in Table C.1.
Although most image/text models and ViT-G/14 JFT showed a higher probing odd-one-out accuracy compared to self-supervised models or models trained on ImageNet, zero-shot odd-one-out accuracy was somewhat less consistent.
For dimension 10, ResNets from Kornblith et al.
[36], trained with a cosine softmax objective, were the best zero-shot performing models, whereas image/text models’ zero-shot performance were among the worst.
For dimension 4, an animal-related concept, models pretrained on ImageNet clearly showed the worst performance, whereas this concepts seems to be well represented in image/text models.
After linear probing, results became less ambiguous.
For almost every human concept, image/text models and ViT-G/14 JFT were the best human aligned models, whereas both AlexNet and EfficientNets achieved the lowest per-concept odd-one-out accuracies.
This difference between image/text models and ViT-G/14 JFT and the other ImageNet-prerained models was particularly apparent for dimension 17 which summarizes sports-related objects.
For this dimension, we observed a large performance gap after linear probing between image/text models and ViT-G/14 JFT and all remaining models.
Analogously, in Appendix G we perform the
Discussion
In this work, we evaluated the alignment of neural network representation with human concept spaces through performance in an odd-one-out task.
Before discussing our findings, we want to address limitations of our work. One obvious limitation is the fact that we did not consider non-linear transformations. It is possible that there exist families of simple non-linear transformations that can provide better alignment for the networks we investigate. We plan to investigate such transformations more thoroughly in future work.
Another limitation relates to the use of pretrained models for our experiments. These models have been trained with a variety of objectives and regularization strategies. We have mitigated this limitation by comparing controlled subsets of models in Figure 3. Nevertheless, we can draw the following conclusions from our findings.
First, scaling ImageNet models does not lead to better alignment of their representations with human similarity judgments.
Differences in human alignment across ImageNet models are mainly attributable to the objective function with which a model was trained, whereas architecture and model capacity are both insignificant.
Second, models trained on image/text or more diverse data achieve much better alignment than
ImageNet models.
Albeit not consistent for zero-shot odd-one-out accuracy, this is clear in both linear probing and regression results.
Third, good representations of concepts that are important to human similarity judgments can be recovered from neural network representation spaces.
However, representations of less important concepts, such as sports and royal objects, are more difficult to recover. How can we train neural networks that achieve better alignment with human concept spaces? Although our results indicate that large, diverse datasets improve alignment, all image/text and JFT models we investigate all attain probing accuracies of 57-58%. By contrast, VICE representations achieve 64%, and a Bayes-optimal classifier achieves 67%.
Since our image/text models are trained on datasets of varying sizes (400M to 6.6B images) but achieve similar alignment, we suspect that further scaling of dataset size is unlikely to close this gap.
To obtain substantial improvements, it may be necessary to incorporate additional forms of supervision when training the representation itself.
Benefits of improving human/machine alignment may extend beyond accuracy on our triplet task, to transfer and retrieval tasks where it is important to capture human notions of similarity. Acknowledgments LM and RV acknowledge support by the Federal Ministry of Education and Research (BMBF) for the Berlin Institute for the Foundations of Learning and Data (BIFOLD) (01IS18037A). We thank Robert Geirhos and Andrew Lampinen for their helpful comments on earlier versions of the manuscript
A Related Work
Most work comparing neural networks with human behavior has focused on the errors made during image classification.
Although ImageNet-trained models appear to make very different errors than humans [52, 19, 20], models trained on larger datasets produce more consistent errors [20], consistent with our findings here.
Compared to humans, ImageNet-trained models perform worse on distorted images [54, 14, 31, 17] and rely more heavily on texture cues and less on object shapes [18, 4], although reliance on texture can be mitigated through data augmentation [18, 30, 41], adversarial training [20], or larger datasets [8].
Previous work has also compared human and machine semantic similarity judgments, albeit using smaller sets of images and models than we explore here.
Jozwik et al.
[33] measured the similarity of AlexNet and VGG-16 representations to human similarity judgments of 92 object images inferred from a multi-arrangement task [37].
Peterson et al.
[48] compared representations of five different neural network architectures to similarity judgments of six different sets of 120 images, obtained by asking subjects to rate the similarities of pairs of images on a scale from 0 to 10.
Like our work, they measure both zero-shot alignment and alignment after a linear transformation, although they consider only diagonal matrices.
Aminoff et al.
[3] showed that, across 11 neural networks, representations of contextually associated objects (e.g., bicycles and helmets) are more similar than those of non-associated objects, and the degree of similarity correlates with both human ratings and reaction times.
Greene et al.
[22] demonstrated that ImageNet-trained convolutional neural networks do not capture human similarity judgments related to actions (“functions”).
Other studies have focused on perceptual similarity rather than semantic similarity, where the task measures perceived similarity between a reference image and a distorted version of that reference image [50, 73], rather than between distinct images as in our task.
Whereas the representations best aligned with human perceptual similarity are obtained from intermediate layers of small architectures [6, 73, 11, 40], the representations best aligned with our odd-one-out judgments are obtained at final model layers, and architecture has little impact.
Our work fits into a broader literature examining relationships between in-distribution accuracy of image classification and other model quality measures, such as accuracy on out-of-distribution data and downstream accuracy when transferring the model.
Out-of-distribution accuracy correlates nearly linearly with accuracy on the training distribution [53, 67, 43], although certain forms of data augmentation can improve accuracy under some distribution shifts without an accompanying improvement in in-distribution accuracy [29].
When comparing the transfer learning performance of different architectures trained with similar settings, accuracy on the pretraining task correlates well with accuracy on the transfer tasks [35], although differences in regularization, training objective, and hyperparameters can have a substantial impact on linear transfer accuracy even if the impact on pretraining accuracy is small [35, 36, 1].
In our study, we find that the training objective has a significant impact, as it does for linear transfer.
However, in contrast to previous observations regarding out-of-distribution generalization and transfer, we find that better-performing architectures do not achieve greater human alignment.
B Experimental details B.1 Linear probing
Initialization We initialized the transformation matrix W ∈ R p × p used in Equation 2 with a temperature scaled identity matrix τ I ∈ R p × p such that W :=
τ I at the beginning of the optimization process.
τ is model-specific and was found via grid search, minimizing the expected calibration error (ECE) [24].
Details on temperature scaling are described in B.2.
Training We optimized the transformation matrix W via gradient descent, using Adam [34] with a learning rate of η = 0 .
We performed a grid-search over the learning rate η , where η ∈ { 0 . 0001 , 0 . 001 , 0 . 01 } and found 0 .
001 to work best for all models in Table C.1.
We trained the linear probe for a maximum of 100 epochs and stopped the optimization process early whenever the generalization performance did not change by a factor of 0 . 0001 for T = 10 epochs.
Cross-validation To obtain a minimally biased estimate of the odd-one-out accuracy of a linear probe, we performed k -fold CV over objects rather than triplets.
We partitioned the m objects into two disjoint sets for train and test triplets.
Algorithm 1 demonstrates how object partitioning was performed for each of the k folds.
Input: ( D , m )
▷ Here, D :=
( { a s , b s } , { i s , j s , k s } ) ns
=1 and m is the number of objects [ m ] = { 1 , . . .
, m } ▷ | [ m ] | = m
O train ∼ U ([ m ])
▷ Sample a number of train objects uniformly at random without replacement O test :=
[ m ] \ O train ▷ Test objects are the remaining objects
D train := {} ▷ Initialize an empty set for the train triplets
D test := {} ▷ Initialize an empty set for the test triplets for s ∈ { 1 , . . . , n } do assignments ≜ list( )
▷ For each triplet initialize an empty list to control object assignments for x ∈ { i s , j s , k s } do if ( x ∈ O train ) then assignment ≜ “train” else assignment ≜ “test” end if assignments ← assignment ▷ Append current assignment to the list of assignments end for if (len(set( assignments ))
̸ = 1) then continue ▷
If not all objects in a triplet belong to the same set of objects, discard triplet else assignment ≜ pop(set( assignments ))
▷ Get object set assignment of current triplet if ( assignment is “train” ) then
D train := D train ∪ D s ▷ Assign current triplet to the train set else
D test := D test ∪ D s ▷ Assign current triplet to the test set end if end if end for
Output: ( D train , D test )
▷ Return both train and test triplet sets Note that the number of train objects that is sampled uniformly at random without replacement from the set of all objects is dependent on k .
We performed a grid-search search over k , where k ∈ { 2 , 3 , 4 , 5 } , and observed that 3 -fold and 4 -fold CV lead to the best linear probing results. Since objects between train and test triplets were not allowed to overlap, loss of data was inevitable (see Algorithm 1).
One can easily see that minimizing the loss of triplet data, comes at the cost of disproportionally decreasing the size of the test set. We decided to proceed with 3 -fold CV in our final experiments since using 2 / 3 of the objects for training and 1 / 3 for testing resulted in a proportionally larger test set than using 3 / 4 for training and 1 / 4 for testing ( ∼ 433 k train and ∼ 54 k test triplets for 3 -fold CV vs. ∼ 616 k train and ∼ 23 k test triplets for 4 -fold CV). In general, the larger a test set, the more accurate the estimate of a model’s generalization performance. To find the optimal strength of the ℓ 2 regularization for each linear probe, we performed a gridsearch over λ for each k value individually.
The optimal λ varied between models, where λ ∈ { 0 . 0001 , 0 .
B.2 Temperature scaling
It is widely known that classifiers trained to minimize cross-entropy tend to be overconfident in their predictions [65, 24, 55], which is in stark contrast to the high-entropy predictions of VICE. We found it helpful to initialize the transformation matrices for the probing experiments using a temperature parameter, as described in Appendix B.1. For this purpose, we performed temperature scaling [24] on the model outputs for THINGS and searched over the scaling parameter τ for each model. In particular, we considered temperature-scaled predictions
[Equation]
where we multiply S in Equation 1 by a constant τ > 0 and S i,j is the inner product of the model representations for images i and j , i.e. the zero-shot similarities.
There are several conceivable criteria that could be minimized to find the optimal scaling parameter τ from a set of candidates.
For our analyses we considered the following,
• Average Jensen-Shannon (JS) distance between model zero-shot probabilities and VICE probabilities over all triplets • Average Kullback-Leibler divergence (KLD) between model zero-shot probabilities and VICE probabilities over all triplets • Expected Calibration Error (ECE) [24].
The ECE is defined as follows.
Let D =
( { a s , b s } , { i s , j s , k s } ) ns =1 be the set of triplets and human responses from Hebart et al.
For a given triplet { i, j, k } and similarity matrix S we define confidence as conf
[Equation]
This corresponds to the expected accuracy of the Bayes classifier for that triplet according to the probability model from S with Equation 1.
We define B m ( S ) to be those training triplets where
[Equation]
For a similarity matrix, S , and a set of triplets with responses, D ′ ⊂ D , we define acc ( D ′ , S ) to be the portion of triplets in D ′ correctly classified according to the highest similarity according to S .
Finally for a set of triplets D ′ ⊂ D and similarity matrix S we define conf( D ′ ) to be the average confidence over that set (triplet responses are simply ignored).
The ECE is defined as
[Equation]
Intuitively, the ECE is low if for each subset B m ( τ S ) the model’s accuracy and its confidence are near each other.
A model will be well-calibrated if its confidence in predicting the odd-one-out in a triplet corresponds to the probability that this prediction is correct.
Of the three considered criteria, ECE resulted in the clearest optima when varying τ , whereas KLD plateaued with increasing τ and JS distance was numerically unstable, most probably because the model output probabilities were near zero for some pairs, which may result in very large JS distance.
For all models, we performed a grid-search over τ ∈ { 1 · 10 0 , 7 . 5 · 10 − 1 , 5 · 10 − 1 , 2 . 5 · 10 − 1 , 1 · 10 − 1 , 7 . 5 · 10 − 2 , 5 · 10 − 2 , 2 . 5 · 10 − 2 , 1 · 10 − 2 , 7 . 5 · 10 − 3 , 5 · 10 − [Table]
B.3 Linear regression
Cross-validation We used ridge regression, that is ℓ 2 -regularized linear regression, to find the transformation matrix A j, : and bias b j that result in the best fit.
We employed nested k -fold CV for each of the d VICE dimensions.
For the outer CV we performed a grid-search over k , where k ∈ { 2 , 3 , 4 , 5 } , similarly to how k -fold CV was performed for linear probing (see B.1).
For our final experiments, we used 5 -fold CV to obtain a minimally biased estimate for the R 2 score of the regression fit.
For the inner CV, we leveraged leave-one-out CV to determine the optimal α for
Equation 3 using RidgeCV from Pedregosa et al.
We performed a grid search over α , where α ∈ { 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000 , 10000 , 100000 , 1000000 } .
C Models
[Figure]
First, we evaluate supervised models trained on ImageNet [57], such as AlexNet [38], various VGGs [61], ResNets [25], EfficientNets [66], ResNext models [69], and Vision Transformers (ViTs) trained on
ImageNet-1K[15] or ImageNet-21K [62] respectively.
Second, we analyze recent state-of-the-art models trained on image/text data, CLIPRN & CLIP-ViT [51], ALIGN [32] and BASIC-L [49].
Third, we evaluate self-supervised (SSL) models that were trained with a contrastive learning objective such as
SimCLR [10] and MoCo [26], recent
SSL models that were trained with a non-contrastive learning objective (no negative examples), BarlowTwins [71], SwAV [9], and VICReg [5], as well as earlier SSL, non-Siamese models, Jigsaw [46], and Rotnet [21].
Last, we evaluate the largest available ViT [72], trained on the proprietary JFT-3B image classification dataset, which consists of approximately three billion images belonging to approximately 30,000 classes [72].
See Table C.1 for further details regarding the models used.
Figure C.1 shows the odd-one-out accuracy as a function of layer depth in a neural network for a few different network architectures.
Later layers generally perform better which is why we performed our analyses exclusively for the logits or penultimate/embedding layers of the models in Table C.1.
[Table]
D CIFAR -100 triplet task
In a similar vein to the THINGS triplet task, we constructed a reference triplet task from the CIFAR -100 dataset [39].
To show pairs of images that are similar to each other, but do not depict the same object, we leverage the 20 coarse classes of the dataset rather than the original fine-grained classes.
For each triplet, we sample two images from the same and an one oddone-out image from a different coarse class.
We restrict ourselves to examples from the CIFAR100 train set and exclude the validation set.
We randomly sample a total of 50 , 000 triplets which is equivalent to the size of the original train set.
Figure D.1 shows an example triplet for this task.
E Linear probing
In the left plot of Figure E.1, we show probing odd-one-out accuracy as a function of ImageNet accuracy for all models in Table C.1.
Similarly to the findings depicted in Figure 2, we observe a low Pearson correlation coefficient ( r = 0 . 213 ) between ImageNet accuracy and probing odd-oneout accuracy.
As a reference, here we show again ImageNet accuracy as a function of zero-shot odd-one-out accuracy on the CIFAR -100 coarse triplet task.
In Figure E.2 we compare probing oddone-out accuracy with zero-shot odd-one-out accuracy for models pretrained on ImageNet-1K or ImageNet-21K.
We observe a strong positive correlation of r = 0 . 963 between probing odd-one-out and zero-shot odd-one-out accuracy.
[Figure]
os h o t o dd - o n e - o u t a cc u r a c
[Figure]
[Figure]
[Figure]
F Human alignment is concept specific
To examine how well neural nets represent human concepts [Figure] we partitioned the original triplet dataset
D into two sets D ∗ and D † , with D ∗ containing triplets correctly predicted by VICE and D † containing those which are not.
The triplets in D † mostly have high entropy, i.e., chosen odd-one-out is not consistent for humans.
The triplets in D † are not used in the following analysis.
We further partitioned D ∗ into 45 subsets according to the 45 VICE dimensions, D ∗ 1 , . . .
, D ∗ 45 .
A triplet belongs to D ∗ j when the sum of the VICE representations for the two most similar objects in the triplet, x a , x b , attains its maximum in dimension j , i.e. j = arg max j ′ x a,j ′ + x b,j ′ .
G Linear regression G.1 Overall performance
In Figure G.1, we compare odd-one-out accuracies after linear probing with zero-shot odd-one-out accuracies and probing odd-one-out accuracies for logits vs. embedding layers of ImageNet models.
The results are consistent with the results from linear probing shown in Figure 4.
G.2 Can human concepts be recovered via linear regression?
In addition to linear probing, we performed ℓ 2 -regularized linear regression to examine models’ ability to predict VICE dimension.
This analysis helped us to further understand whether human concepts can be recovered from a neural network’s representation space.
Here, for each of therepresentation dimensions, j , from VICE, we minimized the following least-squares objective [Equation]
[Equation]
where Y i,j is the value of the j th VICE dimension for image i , x i is the neural network representation of image i , and α j > 0 is a regularization hyperparameter.
Each dimension was optimized separately with α j selected via CV using grid search (details are in Appendix B.3).
[Figure]
e ss i o n
[Figure]
dd -
[Figure] [Figure]
u t a cc u
[Figure]
a c y
[Figure] [Figure]
The results from this analysis corroborate the findings from § 3.2: models trained on image/text data and ViT-G/14 JFT consistently provided the best fit for VICE dimensions, while AlexNet and EfficientNets showed the poorest regression performance.
Furthermore, we investigated whether the recovered VICE dimensions show better alignment than the original network embeddings.
All models were evaluated on the THINGS triplet task using a similarity matrix S with S ij :=
( Ax i + b ) T ( Ax j + b ) , where A and b are obtained by stacking the optimizers from Equation 3, so
Ax + b is a linear regression from a neural network representation to the VICE representation.
In Figure G.2, we compare odd-one-out accuracies after linear probing and regression respectively.
The two performance measures are highly correlated for both the embedding ( r = 0 . 960 ) and logits ( r = 0 .
966) layers.
Note that odd-one-out accuracies are slightly higher for regression.
We hypothesize that this is due to VICE being trained on all objects in the data so the transformation matrix learned in linear regression indirectly has access to all objects opposed to the transformation matrix learned during probing.
Moreover, 2/3 of the objects were used for training the linear probe, whereas 4/5 of the objects were used to fit linear regression.
Figure G.3 shows the R 2 score for fitting a the same subset of VICE dimensions used in Figurefrom embedding-layer representations.
[Figure]
Dimension 4 / Animal-related
[Figure]
i T - J F T A li g n R e s N e t S S L V i T -
I N I n c e p t i o n N a s N e t V G G M o b il e N e t D e n s e N e t R e s N e x t
A l e x N e t
E ff i c
i e n t N e t
Dimension 10 / Vehicles
V i T - J F T B a s i
c
A li g n C L I P V
i T - I N R e
s N e t
R e s N e x t N a s N e t
E ff i c
i e n t N e t D e n s e N e t
I n c e p t i o n S S L V G G M o b il e N e t
A l e x N e t
Dimension 17 / Sports