Original: Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity
Revised: Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity

Original: Abstract
Revised: Abstract

Original: Deep-learning vision models have shown intriguing similarities and differenceswith respect to human vision.
Revised: Deep-learning vision models have shown intriguing similarities and differences with respect to human vision.

Original: We investigate how to bring machine visual representations into better alignment with human representations.
Revised: We investigate how to bring machine visual representations into better alignment with human representations.

Original: Human representationsare often inferred from behavioral evidence such as the selection of an image mostsimilar to a query image.
Revised: Human representations are often inferred from behavioral evidence such as the selection of an image most similar to a query image.

Original: We ﬁnd that with appropriate linear transformationsof deep embeddings, we can improve prediction of human binary choice on adata set of bird images from 72% at baseline to 89%.
Revised: We ﬁnd that with appropriate linear transformations of deep embeddings, we can improve prediction of human binary choice on a data set of bird images from 67.8% at baseline to 90.3%.

Original: We hypothesized that deepembeddings have redundant, high (4096) dimensional representations; however,reducing the rank of these representations  results in a loss of explanatory power.
Revised: We hypothesized that deep embeddings have redundant, high (4096) dimensional representations; reducing the rank of these representations to 2048 results in no loss of explanatory power.

Original: We hypothesized that the dilation transformation of representations explored in pastresearch is too restrictive, and indeed we found that model explanatory power canbe signiﬁcantly improved with a more expressive linear transform.
Revised: We hypothesized that the dilation transformation of representations explored in past research is too restrictive, and indeed we ﬁnd that model explanatory power can be signiﬁcantly improved with a more expressive linear transform.

Original: Most surprisingand exciting, we found that, consistent with classic psychological literature, humansimilarity judgments are asymmetric: the similarity of X to Y is not necessarilyequal to the similarity of Y to X , and allowing models to express this asymmetryimproves explanatory power.
Revised: Most surprising and exciting, we ﬁnd that, consistent with classic psychological literature, human similarity judgments are asymmetric: the similarity of X to Y is not necessarily equal to the similarity of Y to X , and allowing models to express this asymmetry improves explanatory power.

Original: Although deep-learning vision models can sometimes predict aspects of human vision [e.g., 4, 5, 15] their behavior often contrasts sharply with human expectations.
Revised: Although deep-learning vision models can sometimes predict aspects of human vision [e.g., 4, 5, 17, 6, 8] their behavior often contrasts sharply with human expectations.

Original: For instance, small perturbations that are imperceptible to humans can dramatically affect model classiﬁcation decisions [7]; and texture and local image features drive classiﬁers [1, 6] whereas humans are more strongly inﬂuenced by Gestalt shape.
Revised: For instance, small perturbations that are imperceptible to humans can dramatically affect model classiﬁcation decisions [9]; and texture and local image features drive classiﬁers [1, 7] whereas humans are more strongly inﬂuenced by Gestalt shape.

Original: Given that differences exist in how humans and machines represent the world, ourgoal is to develop techniques that bring their representations into better correspondence.
Revised: Given that differences exist in how humans and machines represent the world, our goal is to develop techniques that bring their representations into better correspondence.

Original: This goal isimportant for two reasons.
Revised: This goal is important for two reasons.

Original: First, human vision is robust and visual representations contain a wealth ofinformation about objects and their properties.
Revised: First, human vision is robust and visual representations contain a wealth of information about objects and their properties.

Original: Bringing representations into alignment might expandthe range of tasks for which deep nets are useful [e.g., 10, 16, 23].
Revised: Bringing representations into alignment might expand the range of tasks for which deep nets are useful [e.g., 12, 18, 26].

Original: Second, if the correspondenceis strong, deep nets can serve as a human surrogate for prediction and optimization, allowing us toefﬁciently determine, say, the best training procedures for people [2, 18, 14, 19].
Revised: Second, if the correspondence is strong, deep nets can serve as a human surrogate for prediction and optimization, allowing us to efﬁciently determine, say, the best training procedures for people [2, 20, 16, 22].

Original: Let’s be more speciﬁc about the representations that need to be aligned.
Revised: Let’s be more speciﬁc about the representations that need to be aligned.

Original: In a deep net trained toclassify images, the representation in the penultimate layer (prior to the softmax layer) serves asa deep embedding of the image.
Revised: In a deep net trained to classify images, the representation in the penultimate layer (prior to the softmax layer) serves as a deep embedding of the image.

Original: This representation necessarily contains the features essential fordiscriminating object categories.
Revised: This representation necessarily contains the features essential for discriminating object categories.

Original: One might hope to align this representation with the activity patternin higher cortical areas of the human brain, i.e., a neural embedding , but it is not feasible to readout large-scale brain activation at a sufﬁciently ﬁne spatial and temporal resolution.
Revised: One might hope to align this representation with the activity pattern in higher cortical areas of the human brain, i.e., a neural embedding , but it is not feasible to read out large-scale brain activation at a sufﬁciently ﬁne spatial and temporal resolution.

Original: Instead, onemight hope to align psychological embeddings —a representation of the features essential for humanclassiﬁcation, judgment, decision making, and information processing.
Revised: Instead, one might hope to align psychological embeddings —a representation of the features essential for human classiﬁcation, judgment, decision making, and information processing.

Original: A common method to obtain psychological embeddings requires collecting similarity ratings betweenpairs of items in a domain and then inferring an embedding in which more similar pairs are closer in the embedding space than less similar pairs.
Revised: A common method to obtain psychological embeddings requires collecting similarity ratings between pairs of items in a domain and then inferring an embedding in which more similar pairs are closer in the embedding space than less similar pairs.

Original: Multidimensional scaling (MDS) [21] has been used for over half a century to obtain psychological embeddings for a ﬁxed set of items whose pairwisesimilarity matrix is provided.
Revised: Multidimensional scaling (MDS) [24] has been used for over half a century to obtain psychological embeddings for a ﬁxed set of items whose pairwise similarity matrix is provided.

Original: Even at a large scale, it can obtain low-dimensional interpretableembeddings that generalize to behavioral tasks [8].
Revised: Even at a large scale, it can obtain low-dimensional interpretable embeddings that generalize to behavioral tasks [10].

Original: Although MDS can be used given partial or noisysimilarity matrices, it is not productive in the sense that it allows one to predict representations andsimilarities only for items contained in the original similarity matrix.
Revised: Although MDS can be used given partial or noisy similarity matrices, it is not productive in the sense that it allows one to predict representations and similarities only for items contained in the original similarity matrix.

Original: Ideally, one desires an open-setmethod, not one that works only for the ﬁxed, previously rated set.
Revised: Ideally, one desires an open-set method, not one that works only for the ﬁxed, previously rated set.

Original:  Background and Related Research
Revised:  Background and Related Research

Original: Toward the goal of being able to obtain a psychological embedding for any image, methods havebeen proposed that leverage human similarity judgments in conjunction with deep nets.
Revised: Toward the goal of being able to obtain a psychological embedding for any image, methods have been proposed that leverage human similarity judgments in conjunction with deep nets.

Original: These netshave the advantage over MDS that they can in principle embed novel images.
Revised: These nets have the advantage over MDS that they can in principle embed novel images.

Original: Sanders and Nosofsky [19] trained a fully-connected net to re-map from a deep embedding of apretrained classiﬁer to an MDS representation.
Revised: Sanders and Nosofsky [22] trained a fully-connected net to re-map from a deep embedding of a pretrained classiﬁer to an MDS representation.

Original: They found that the resulting re-mapping generalizedwell to images held out from the re-mapping training set.
Revised: They found that the resulting re-mapping generalized well to images held out from the re-mapping training set.

Original: While this approach demonstrates thatpsychological embeddings can be extracted from deep embeddings, the approach is limited in that itproduces a representation that is no richer than the MDS embedding.
Revised: While this approach demonstrates that psychological embeddings can be extracted from deep embeddings, the approach is limited in that it produces a representation that is no richer than the MDS embedding.

Original: The dimensionality of MDSembeddings is limited by the quantity of human judgment data available; deep embeddings are not.
Revised: The dimensionality of MDS embeddings is limited by the quantity of human judgment data available; deep embeddings are not.

Original: Peterson, Abbott, and Grifﬁths [14] bypassed the MDS embedding and used a deep embedding todirectly predict human similarity judgments.
Revised: Peterson, Abbott, and Grifﬁths [16] bypassed the MDS embedding and used a deep embedding to directly predict human similarity judgments.

Original: These judgments, made on image pairs using a 0–10scale with larger values indicating greater similarity, were placed into a symmetric similarity matrix
Revised: These judgments, made on image pairs using a 0– scale with larger values indicating greater similarity, were placed into a symmetric similarity matrix

Original: S , where element s ij is the judgment for image pair i and j .
Revised: S , where element s ij is the judgment for image pair i and j .

Original: The matrix is modeled with ˆ S , whoseelements are deﬁned as:
Revised: The matrix is modeled with ˆ S , where

Original: [Equation]
Revised: [Equation]

Original: ) where z i and z j are the deep embeddings for images i and j  and W is a diagonal matrix.
Revised: and z i and z j are the deep embeddings for images i and j , and W is a diagonal matrix.

Original: Theparameters of W are obtained by optimization of a squared loss, || S − ˆ S
Revised: The parameters of W are obtained by optimization of a squared loss, || S − ˆ S

Original: || 2 , with an L 2 regularizerto prevent overﬁtting.
Revised: || 2 , with an L 2 regularizer to prevent overﬁtting.

Original: Peterson et al.
Revised: Peterson et al.

Original: found that the best ﬁts to human data are obtained by a VGGarchitecture [20], whose embedding layer is 4096 dimensional.
Revised: found that the best ﬁts to human data are obtained by a VGG architecture [23], whose embedding layer is 4096 dimensional.

Revised: Peterson et al.

Revised: placed no constraint on the sign of the elements of W .

Original: With non-negative elements, W can be decomposed as V T V , and Peterson et al.
Revised: However, with non-negative elements, W can be decomposed as V T V , and Peterson et al.

Original: ’s method can beviewed as a form of deep metric learning [9, 11].
Revised: ’s method can be viewed as a form of deep metric learning [11, 13].

Original: That is, the similarity function can be interpretedas computing a dot product of linearly transformed embeddings, i.e., ˆ s ij =
Revised: That is, the similarity function can be interpreted as computing a dot product of linearly transformed embeddings, i.e., ˆ s ij =

Original: ( V z i ) T ( V z j ) .
Revised: ( V z i ) T ( V z j ) .

Original: In thiscase, the linear transform rescales individual features (vector elements) of the embedding.
Revised: In this case, the linear transform rescales individual features (vector elements) of the embedding.

Original: Sucha transform makes the most sense if these features can be ascribed psychological meaning: whencomparing vectors, the rescaling permits some features to matter more, some less.
Revised: Such a transform makes the most sense if these features can be ascribed psychological meaning: when comparing vectors, the rescaling permits some features to matter more, some less.

Original: However, the basisused for representing the embedding is completely arbitrary: any rotation of the basis is a functionallyequivalent solution to the classiﬁer (because a linear transform of the embedding is performed in thesoftmax output layer).
Revised: However, the basis used for representing the embedding is completely arbitrary: any rotation of the basis is a functionally equivalent solution to the classiﬁer (because a linear transform of the embedding is performed in the softmax output layer).

Original: Consequently, we question whether it is well motivated to restrict transforms todilating a representation that lies in an arbitrary basis.
Revised: Consequently, we question whether it is well motivated to restrict transforms to dilating a representation that lies in an arbitrary basis.

Original: Peterson et al.
Revised: Peterson et al.

Original: likely made this sensible choicebecause a full W would have 16M parameters and would be underconstrained by the relatively smallnumber of human judgments.
Revised: likely made this choice because a full W would have 16M parameters and would be underconstrained by the relatively small number of human judgments.

Original: We describe a potential solution to this dilemma that both gives theembedding a non-arbitrary basis—hopefully one with psychological validity—and allows us to varythe number of free parameters in the model.
Revised: We describe a potential solution to this dilemma that both gives the embedding a non-arbitrary basis—hopefully one with psychological validity—and allows us to vary the number of free parameters in the model.

Original: We have two further concerns with Peterson et al.
Revised: We have two further concerns with Peterson et al.

Original: ’s method which our solution addresses.
Revised: ’s method which our solution addresses.

Original: First, Peterson et al.
Revised: First, Peterson et al.

Original: z -score normalize the embeddings from the pretrained model before using the embeddingsto compute similarity.
Revised: z -score normalize the embeddings from the pretrained model before using the embeddings to compute similarity.

Original: Variance normalization does not matter because any such normalization canbe inverted via W .
Revised: Variance normalization does not matter because any such normalization can be inverted via W .

Original: However, the zero centering of z -scoring alters the angles between embeddingvectors, and those angles are not arbitrary: the original classiﬁer uses these angles to compute softmaxprobabilities.
Revised: However, the zero centering of z -scoring alters the angles between embedding vectors, and those angles are not arbitrary: the original classiﬁer uses these angles to compute softmax probabilities.

Original: Second, Peterson et al.
Revised: Second, Peterson et al.

Original: use absolute similarity ratings for model training.
Revised: use absolute similarity ratings for model training.

Original: Such ratingsare subject to sequential dependencies [13], reducing the signal they convey.
Revised: Such ratings are subject to sequential dependencies [15], reducing the signal they convey.

Original: Relative judgments—ofthe form ‘is X more similar to Y than to Z ?’
Revised: Relative judgments—of the form ‘is X more similar to Y than to Z ?’

Original: —tend to be more reliable, despite outwardly conveyingless information [3, 12, 24].
Revised: —tend to be more reliable, despite outwardly conveying less information [3, 14, 27].

Original:  Methodology2.1 Data set
Revised:  Methodology 2.1 Data set

Original: We used a previously collected data set of similarity judgments for bird images [18].
Revised: We used a previously collected data set of similarity judgments for bird images [20].

Original: The image setcontains four bird families (Orioles, Warblers, Sparrows, and Cardinals), four distinct species within each family, and thirteen distinct images of each species.
Revised: The image set contains four bird families (Orioles, Warblers, Sparrows, and Cardinals), four distinct species within each family, and thirteen distinct images of each species.

Original: (Examples are presented in Figure 2 of[18].)
Revised: Examples are presented in Figure 2 of [20].

Original: Mechanical Turk participants were shown a query image along with two reference images andchose which reference was most similar to the query.
Revised: Mechanical Turk participants were shown a query image along with two reference images and chose which reference was most similar to the query.

Original: See the left edge of Figure 1 for a sample triple.
Revised: See the left edge of Figure 1 for a sample triple.

Original: The resulting judgment providess a triplet inequality constraint ( TIC ).
Revised: The resulting judgment providess a triplet inequality constraint ( TIC ).

Original: Some participants were shownthe query with eight reference images and were asked to choose the two most similar.
Revised: Some participants were shown the query with eight reference images and were asked to choose the two most similar.

Original: Data fromthese trials provide 12 TICs: each of the two chosen references is more similar to the query than eachof the six non-chosen references.
Revised: Data from these trials provide 12 TICs: each of the two chosen references is more similar to the query than each of the six non-chosen references.

Original: The complete data set consisted of 112,784 TICs.
Revised: The complete data set consisted of 112,784 TICs.

Original: [Figure]
Revised: 2.2 Models to be evaluated

Original: Figure 1 sketches the structure of our approach.
Revised: Figure 1 sketches the structure of our approach.

Original: To obtain deep embeddings, we use a headless
Revised: To obtain deep embeddings, we use a headless

Original: VGG16 classiﬁer pretrained on ImageNet from the Keras library.
Revised: VGG16 classiﬁer pretrained on ImageNet from the Keras library.

Original: The penultimate layer of VGG16has 4,096 units.
Revised: The penultimate layer of VGG has 4,096 units.

Original: To model the TIC, we pass the query and two references through VGG16 andthen compute pairwise similarities between the query and each of the references.
Revised: To model the TIC, we pass the query and two references through VGG16 and then compute pairwise similarities between the query and each of the references.

Original: For a query q andreference r , we generalize the learned similarity function of Equation 1 as follows:
Revised: For a query q and reference r , we generalize the learned similarity function of Equation 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: where f : R 4096 → R k performs dimensionality reduction on the original 4096-dimensional deepembedding.
Revised: where f : R 4096 → R k performs dimensionality reduction on the original 4096-dimensional deep embedding.

Original: Variants of this model are speciﬁed via choice of f ( . ) , k , and the constraints placed on
Revised: Variants of this model are speciﬁed via choice of f ( . ) , k , and the constraints placed on

Original: W .
Revised: W .

Original: We explore these speciﬁc constraints on W :
Revised: We explore these speciﬁc constraints on W :

Original: • Identity : Use original deep embedding space via W =
Revised: • Identity : Use original deep embedding space via W =

Original: I .
Revised: I .

Original: This case serves as a baseline.• Regularized Diagonal :
Revised: This case serves as a baseline. • Diagonal :

Original: Rescale the original embedding with a diagonal matrix and impose an L 2penalty on the elements.
Revised: Rescale the original embedding with a diagonal matrix.

Original: This constraint on W was used by Peterson et al.
Revised: In contrast to Peterson et al.

Original: (Equation 1). We require the diagonal elements to be non-negative by optimizing for an unconstrained diagonal vector v ∈ R k and using W = diag( | v | ) .
Revised: , We require the diagonal elements to be non-negative by optimizing for an unconstrained diagonal vector v ∈ R k and using W = diag( | v | ) .

Original: • Diagonal : Same as the previous method except without regularization.•
Revised: We used this constraint because it seems antithetical to posit that greater alignment of representations should reduce similarity.

Original: Symmetric : Require symmetry of the k × k matrix, which allows us to interpret the similarityfunction as applying an arbitrary linear transform to each embedding and then computing theirdot-product similarity (see Introduction).
Revised: Symmetric : Require symmetry of the k × k matrix, which allows us to interpret the similarity function as applying an arbitrary linear transform to each embedding and then computing their dot-product similarity (see Introduction)

Original: We optimize over an unconstrained matrix, V ∈ R k × k ,where W = V TV .
Revised: We optimize over an unconstrained matrix, V ∈ R k × k , where W = V T

Original: • Unconstrained : We optimize directly over an unconstrained W ∈ R
Revised: V . This approach is related to that of Ryali et al. [21], who learn the parameters of a

Revised: covariance matrix to compute the Mahalanobis distance between representations.

Revised: We question

Revised: whether a Mahalanobis distance is the appropriate metric when deep embeddings are trained and used to classify via dot-product softmax functions.

Revised: • Unconstrained : We optimize directly over an unconstrained W ∈ R

Original: k × k .
Revised: k × k .

Original: In picking k and f ( . ) , we have two goals.
Revised: In picking k and f ( . ) , we have two goals.

Original: First, we wish to reduce the number of free parametersin our models to avoid overﬁtting the training data.
Revised: First, we wish to reduce the number of free parameters in our models to avoid overﬁtting the training data.

Original: Second, we wish to impose a basis that is lessarbitrary than that of the original deep embedding.
Revised: Second, we wish to impose a basis that is less arbitrary than that of the original deep embedding.

Original: As we argued earlier, the basis obtained fromtraining VGG is arbitrary; any rank-preserving linear transform is an equivalent solution given thatthis transform could be inverted in the softmax layer to achieve the same output.
Revised: As we argued earlier, the basis obtained from training VGG is arbitrary; any rank-preserving linear transform is an equivalent solution given that this transform could be inverted in the softmax layer to achieve the same output.

Original: If W is constrainedto be diagonal, it seems desirable for the dimensions rescaled by W to have some psychologicalreality.
Revised: If W is constrained to be diagonal, it seems desirable for the dimensions rescaled by W to have some psychological reality.

Original: To achieve these two goals in the simplest manner possible, we treat f as the projection of thedeep embedding onto the top k principal components of the embedding space.
Revised: To achieve these two goals in the simplest manner possible, we treat f as the projection of the deep embedding onto the top k principal components of the embedding space.

Original: PCA is performed onthe embeddings of 23,400 images of the 18 bird classes in the ImageNet training set. Note that theseimages are distinct from those used for similarity assessment. [Figure]
Revised: PCA is performed on

Original: [Figure]
Revised: [Figure]

Original: (choose reference 1 ) human triplet judgments of the form‘Is the query more similar to reference 1 or 2?’
Revised: of the form the query more similar to reference 1 or 2?’

Original: After embedding each image, pairwise similarities are computed and the relative similarity determines the probability of choosing one reference or the other.
Revised: After embedding each image, pairwise similarities are computed and the relative similarity determines the probability of choosing one reference or the other.

Original: [Figure]
Revised: the embeddings of 23,400 images of the 18 bird classes in the ImageNet training set. Note that these images are distinct from those used for similarity assessment.

Original: To obtain a prediction for the human judgment, we follow a long tradition in choice modeling andassume a logistic function of the relative similarities for query q and references r 1 and r 2 :
Revised: To obtain a prediction for the human judgment, we follow a long tradition in choice modeling and assume a logistic function of the relative similarities for query q and references r 1 and r 2 :

Original: Pr
Revised: Pr(choose ref c | q, r 1 , r 2 ) = logistic(ˆ s q,r c − ˆ s q,r 3 − c ) . Note that because of the linearity of f and ˆ s , Pr(choose ref c |

Original: [Equation]
Revised: [Equation]

Original: choose ref
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: logistic
Revised: logistic

Original: [Equation]

Original: [Equation]

Original: Note that because of the linearity of f and ˆ s ,

Original: [Equation]
Revised: [Equation]

Original: 2.3 Simulation details
Revised: 2.3 Simulation details

Original: Models are trained to maximize log likelihood of the training triples,
Revised: Models are trained to maximize log likelihood of the training triples,

Original: [Equation]
Revised: [Equation]

Original: ) where c is the index of the reference chosen by the human rater and T is the training set.
Revised: where c is the index of the reference chosen by the human rater and T is the training set.

Original: Five-foldcross-validation was performed, yielding 90k and 22.5k triplets in each fold for the training andvalidation sets, respectively.
Revised: Five-fold cross-validation was performed, yielding 90k and 22.5k triplets in each fold for the training and validation sets, respectively.

Original: We trained models using TensorFlow with an SGD optimizer, with
Revised: We trained models using TensorFlow with an SGD optimizer, with

Original: Nesterov momentum of 0.9, an initial learning rate of 10 − 9 for the Unconstrained model and 10 − 5for all others, with an exponential scale-down of the learning rate by a factor of 0.9 every 40 epochs.
Revised: Nesterov momentum of 0.9, an initial learning rate of 10 − 9 for the Unconstrained model and 10 −for all others, with an exponential scale-down of the learning rate by a factor of 0.9 every 40 epochs.

Original: Although trained with likelihood maximization, we evaluate models on accuracy , deﬁned as theproportion of examples in which the human response matches the most likely model response.
Revised: Although trained with likelihood maximization, we evaluate models on accuracy , deﬁned as the proportion of examples in which the human response matches the most likely model response.

Original: Asweak prevention against overﬁtting, we stopped training when the training accuracy did not improvein the last 10 epochs over the 10 epochs previous.
Revised: As weak prevention against overﬁtting, we stopped training when the training accuracy did not improve in the last 10 epochs over the 10 epochs previous.

Original: A second possible factor mitigating overﬁttingis that optimization of the Unconstrained and Symmetric models with large k often blew up withnot-a-number-errors late in training, which led us to terminate and use the previous valid weights.

Original:  Results
Revised:  Results

Original: Figure 2 shows the outcome of ﬁve-fold cross-validation on our data set of human similarity judgmentsof bird images.
Revised: Figure 2 shows the outcome of ﬁve-fold cross-validation on our data set of human similarity judgments of bird images.

Original: The left and right panels show training and validation set accuracy, respectively.
Revised: The left and right panels show training and validation set accuracy, respectively.

Original: Amodel prediction is scored as correct if the model’s probability of selecting the reference chosen bythe human is greater than 0.5.
Revised: A model prediction is scored as correct if the model’s probability of selecting the reference chosen by the human is greater than 0.5.

Original: Accuracy is plotted as a function of the number of principal-componentloadings included in the deep embedding ( k ), and a separate curve is drawn for each differentconstraint on W that we tested.
Revised: Accuracy is plotted as a function of the number of principalcomponent loadings included in the deep embedding ( k ), and a separate curve is drawn for each different constraint on W that we tested.

Original: We were unable to complete k = 4096 runs by submission time, butthe curves appear to be ﬂattening so we do not expect surprises for k = 4096 .
Revised: The dashed blue line is an implementation of Peterson et al. ’s method: diagonal weights, allowing negative values, the full 4096-element vector, and imposing an L 2 penalty on the weights. To give this model its best shot, we searched for the L 2 regularization coefﬁcient over many orders of magnitude that maximized performance on the validation set (i.e., we cheated to beneﬁt this method); we also did not z -score the embeddings for reasons explained earlier.

Original: Validation performance roughly tracks training performance.
Revised: Validation performance roughly tracks training performance.

Original: However, for the most complex models(Symmetric and Unconstrained W and large k ) there appears to be some overﬁtting: the training curverises faster than the validation curve.
Revised: However, for the most complex models (Symmetric and Unconstrained W and large k ) there appears to be some overﬁtting: the training curve rises faster than the validation curve.

Original: We expected to observe more severe overﬁtting, manifested byadrop in validation performance with k , because the largest models have anearly 40:1 (Unconstrained,k = 2048 ) and 20:1 (Symmetric, k = 2048 ) ratio of free parameters to training examples (TICs). Itappears that overﬁtting is prevented by some combination of two factors: (1) the linear form of the
Revised: We expected to observe more severe overﬁtting, manifested by a

Original: [Figure]
Revised: [Figure]

Original: model offers a sufﬁciently strong constraint on data patterns that can be ﬁt, (2) inconsistency in  the 
Revised: drop in validation performance with k , because the largest models have a nearly 185:1 (Unconstrained, k = 4096 ) and 93:1 (Symmetric, k = 4096 ) ratio of free parameters to training examples (TICs).

Original: human judgments causes TICs to violate transitivity which cannot be ﬁt by the models we consider.
Revised: Contrary to our predictions, the inverted U function is not observed, possibly due to the linear form of the model, which offers a strong constraint on data patterns that can be ﬁt, or to inconsistency in the human judgments that results in violations of transitivity, not ﬁttable by our models.

Original: The validation curves in Figure 2b allow us to draw some strong and intriguing conclusions
Revised: The validation curves in Figure 2b allow us to draw some strong and intriguing conclusions:

Original: • Dilation of the transformed deep-embedding features (blue curves) achieves a signiﬁcantly betterﬁt to the human data than by simply using the deep embedding straight from the VGG16 classiﬁer(grey curve).
Revised: • Dilation of the transformed deep-embedding features (blue curves) achieves a signiﬁcantly better ﬁt to the human data than by simply using the deep embedding straight from the VGG16 classiﬁer (grey curve).

Original: This result replicates the key ﬁnding of Peterson et al.
Revised: This result replicates the key ﬁnding of Peterson et al.

Original: on a new data set and a differentresponse measure.
Revised: on a new data set and a different response measure.

Original: Our work extends Peterson et al.
Revised: Our work extends Peterson et al.

Original: by using the transform f ( . ) and showing thatincreasing the dimensionality of the embedding strictly improves the ﬁt to human data.
Revised: by using the transform f ( . ) and showing that increasing the dimensionality of the embedding strictly improves the ﬁt to human data, and that adding the non-negativity constraint slightly improves results (for k = 4096 ).

Original: • Over the range of L 2 regularizer coefﬁcients we explored, we did not see a beneﬁt for the regularizedmodel (light blue curve) over the corresponding model without regularization (dark blue curve).

Original: This ﬁnding is consistent with our general observation that overﬁtting is not occuring with the classof models we consider and our data set.

Original: • Applying a general linear transform to the deep embedding (purple curve) obtains a better ﬁt tothe human data than a dilation (blue curves).
Revised: • Applying a general linear transform to the deep embedding (purple curve) obtains a better ﬁt to the human data than a dilation (blue curves).

Original: Peterson et al.
Revised: Peterson et al.

Original: did not investigate using the broader class of transform because it seemed likely that overﬁtting would occur.
Revised: did not investigate using the broader class of transform because it seemed likely that overﬁtting would occur.

Original: However we see a strictimprovement in performance with the more complex model, regardless ofk .•
Revised: However we see a strict improvement in performance with the more complex model, regardless of

Original:  Relaxing the symmetry constraint on similarity (i.e., the similarity of A to B does not have to equal the similarity of B to A) improves the ﬁt to human data (red versus purple curve).
Revised: k . • Relaxing the symmetry constraint on similarity (i e., the similarity of A to B does not have to equal the similarity of B to A) improves the ﬁt to human data (red versus purple curve).

Original: Thisﬁnding was most surprising to us, but in retrospect might have been anticipated by the prominent, longstanding ﬁnding that human judgments of similarity cannot be accounted for by the use of an internal psychological distance metric [22].
Revised: This ﬁnding was most surprising to us, but in retrospect might have been anticipated by the prominent, longstanding ﬁnding that human judgments of similarity cannot be accounted for by the use of an internal psychological distance metric [25].

Original: For example, individuals might judge North Korea to be more similar to China (focusing on the leadership) than China is to North Korea (focusing on the size of the country).
Revised: For example, individuals might judge North Korea to be more similar to China (focusing on the leadership) than China is to North Korea (focusing on the size of the country).

Original: To cast this claim in terms of the judgments we are modeling, considertwo different triplets: (query I 1 , references I 2 and I 3 ) and (query I 2 , references I 1{+
+} and I 3 ).
Revised: To cast this claim in terms of the judgments we are modeling, consider two different triplets: (query I 1 , references I 2 and I 3 ) and (query I 2 , references I {+
+} and I 3 ).

Original: Thedeep embedding of I 

Revised: The deep embedding of I

Original: and I 2 are interpreted differently depending on whether they are in the roleof query or reference.
Revised: and I 2 are interpreted differently depending on whether they are in the role of query or reference.

Original: [Figure]
Revised: [Figure]

Revised: with held-out triplets vs. held-out images

Original: In the above results,  accuracy is assessed on heldout triplets (TICs).
Revised: In the above results, human-prediction accuracy is assessed on held-out triplets (TICs).

Original: To evaluate accuracy for held-outimages , we ran validation folds in which we randomlyselected images to hold out such that the training setwas roughly the same size as in our earlier experiment.
Revised: To evaluate accuracy for held-out images , we ran validation folds in which we randomly selected images to hold out such that the training set was roughly the same size as in our earlier experiment.

Original: To best match our earlier experiment, we performﬁve fold validation (sampling with replacement eachfold) for both held-out triplets and held-out imageswith k = 2048 .
Revised: To best match our earlier experiment, we perform ﬁve fold validation (sampling with replacement each fold) for both held-out triplets and held-out images with k = 2048 .

Original: As Figure 3 indicates, models dogeneralize to new images.
Revised: As Figureindicates, models do generalize to new images.

Original: The ranking of models isthe same, but performance does suffer on new images.
Revised: The ranking of models is the same, but performance does suffer on new images.

Original:  Discussion
Revised:  Discussion

Original: Our models of similarity judgment are able to predict89% of human binary choices, suggesting that a deep embedding from a pretrained classiﬁer can beadapted to capture the structure of psychological embeddings of visual images.
Revised: Our models of similarity judgment are able to predict 90.3% of human binary choices, suggesting that a deep embedding from a pretrained classiﬁer can be adapted to capture the structure of psychological embeddings of visual images.

Original: By applying a lineartransform to a deep embedding, we are able to boost the accuracy of prediction from a 72% baselineusing the original embedding.
Revised: By applying a linear transform to a deep embedding, we are able to boost the accuracy of prediction from a 67.8% baseline using the original embedding.

Original: We signiﬁcantly improve on an existing approach in the literature [14],which achieves a prediction accuracy on our data set of only 78%.
Revised: We signiﬁcantly improve on an existing approach in the literature [16], which achieves a prediction accuracy on our data set of only 78.0%.

Original: Our simulations reveal several surprising and intriguing results.
Revised: Our simulations reveal several surprising and intriguing results.

Original: First, we observe that overﬁtting isnot a serious issue for highly overparameterized linear models—models with forty times as manyfree parameters as training data points.
Revised: First, we observe that overﬁtting is not a serious issue for highly overparameterized linear models—models with forty times as many free parameters as training data points.

Original: We are presently investigating whether this ﬁnding is dueto model linearity (which restricts the transformations that can be applied to the deep embedding)or to some fortuitous early-stopping procedure.
Revised: We are presently investigating whether this ﬁnding is due to model linearity (which restricts the transformations that can be applied to the deep embedding) or to some fortuitous early-stopping procedure.

Original: Second, and most notably, we observe a beneﬁt forencoding similarity in form that cannot be expressed in terms of distance metric in the embeddingspace.
Revised: Second, and most notably, we observe a beneﬁt for encoding similarity in form that cannot be expressed in terms of distance metric in the embedding space.

Original: Rather, one item of a pair is treated as an anchor with respect to which the other item iscompared.
Revised: Rather, one item of a pair is treated as an anchor with respect to which the other item is compared.

Original: To the best of our knowledge, researchers in machine learning who model human similarhave done so based on distance metrics that do not permit the sort of asymmetry supported by ourdata [e.g., 10, 17, 24, 23].
Revised: To the best of our knowledge, researchers in machine learning who model human similar have done so based on distance metrics that do not permit the sort of asymmetry supported by our data [e.g., 12, 19, 27, 26].

Original: Considering the role of anchoring and context in choice is a productiveavenue for future research.
Revised: Considering the role of anchoring and context in choice is a productive avenue for future research.

Revised: Acknowledgments and Disclosure of Funding

Revised: We thank Gamaledin Elsayed for helpful feedback on the manuscript.