Original: VICE: Variational Interpretable Concept Embeddings
Revised: VICE: Variational Interpretable Concept Embeddings

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: A central goal in the cognitive sciences is the development of numerical modelsfor mental representations of object concepts.
Revised: A central goal in the cognitive sciences is the development of numerical modelsfor mental representations of object concepts.

Original: This paper introduces Variational
Revised: This paper introduces Variational

Original: Interpretable Concept Embeddings (VICE), an approximate Bayesian method forembedding object concepts in a vector space using data collected from humansin an odd-one-out triplet task.
Revised: Interpretable Concept Embeddings (VICE), an approximate Bayesian method forembedding object concepts in a vector space using data collected from humansin an odd-one-out triplet task.

Original: VICE uses variational inference to obtain sparse,non-negative representations of object concepts with uncertainty estimates for theembedding values.
Revised: VICE uses variational inference to obtain sparse,non-negative representations of object concepts with uncertainty estimates for theembedding values.

Original: These estimates are used to automatically select the dimensionsthat best explain the data.
Revised: These estimates are used to automatically select the dimensionsthat best explain the data.

Original: We derive a PAC learning bound for VICE that can beused to estimate generalization performance or determine sufficient sample size inexperimental design.
Revised: We derive a PAC learning bound for VICE that can beused to estimate generalization performance or determine sufficient sample size inexperimental design.

Original: VICE rivals or outperforms its predecessor, SPoSE, at predicting human behavior in the odd-one-out triplet task.
Revised: VICE rivals or outperforms its predecessor, SPoSE, at predicting human behavior in the odd-one-out triplet task.

Original: Furthermore, VICE’s objectrepresentations are more reproducible and consistent across random initializations
Revised: Furthermore, VICE’s objectrepresentations are more reproducible and consistent across random initializations

Original:  Introduction
Revised:  Introduction

Original: Human knowledge about object concepts encompasses many types of information, including functionor purpose, visual appearance, encyclopedic facts, or taxonomic characteristics.
Revised: Human knowledge about object concepts encompasses many types of information, including functionor purpose, visual appearance, encyclopedic facts, or taxonomic characteristics.

Original: A central questionin cognitive science concerns the representation of this knowledge and its use across different tasks.
Revised: A central questionin cognitive science concerns the representation of this knowledge and its use across different tasks.

Original: One approach to this question is inductive and lets subjects list properties for hundreds to thousandsof objects [27, 9, 6, 19], yielding large lists of responses about the different types of properties.
Revised: One approach to this question is inductive and lets subjects list properties for hundreds to thousandsof objects [27, 9, 6, 19], yielding large lists of responses about the different types of properties.

Original: Thisrepresents objects as vectors of binary properties.
Revised: Thisrepresents objects as vectors of binary properties.

Original: While this approach is agnostic to downstreamprediction tasks it may be biased since subjects may leave out important features and/or mentionunimportant ones.
Revised: While this approach is agnostic to downstreamprediction tasks it may be biased since subjects may leave out important features and/or mentionunimportant ones.

Original: For example, one may forego a general property (e.g., “is an animal”) whileproviding a highly specific fact (e.g., “is found in Florida”).
Revised: For example, one may forego a general property (e.g., “is an animal”) whileproviding a highly specific fact (e.g., “is found in Florida”).

Original: In an alternative, deductive, approach,researchers postulate dimensions of interest and subsequently let subjects place objects in eachdimension.
Revised: In an alternative, deductive approach,researchers postulate dimensions of interest and subsequently let subjects place objects in eachdimension.

Original: Binder et al.
Revised: Binder et al.

Original: [3] employed such an approach and collected ratings for hundreds of objects,verbs, and adjectives.
Revised: [3] employed such an approach and collected ratings for hundreds of objects,verbs, and adjectives.

Original: These ratings were gathered over 65 dimensions, reflecting sensory, motor,spatial, temporal, affective, social, and cognitive experiences.
Revised: These ratings were gathered over 65 dimensions, reflecting sensory, motor,spatial, temporal, affective, social, and cognitive experiences.

Original: Nonetheless, it is desirable to discoverobject representations that are not biased by the conducted behavioral task and whose dimensions areinterpretable without necessitating a priori assumptions about their semantic content.
Revised: Nonetheless, it is desirable to discover object representations that are not biased by the conducted behavioral task and whose dimensions are interpretable without necessitating a priori assumptions about their semantic content.

Original: Recently, Zheng et al.
Revised: Recently, Zheng et al.

Original: [45] and Hebart et al.
Revised: [45] and Hebart et al.

Original: [18] introduced SPoSE, a model of the mental representations of 1,854 objects in a 49-dimensional space.
Revised: [18] introduced SPoSE, a model of the mental represen- tations of 1,854 objects in a 49-dimensional space.

Original: The model was learned from human judgmentsabout object similarity, where subjects were asked to determine an odd-one-out object in randomtriplets of objects.
Revised: The model was learned from human judgments about object similarity, where subjects were asked to determine an odd-one-out object in random triplets of objects.

Original: SPoSE embedded each object in a vector space so that each dimension is nonnegative and sparse (most objects have a zero entry for a given dimension).
Revised: SPoSE embedded each object in a vector space so that each dimension is nonnegative and sparse (most objects have a zero entry for a given dimension).

Original: The authors showed thatthe embedding dimensions of objects were interpretable and subjects could coherently label what the dimensions were “about,” ranging from categorical (e.g., animate, food) to functional (e.g., tool), structural (e.g., made of metal or wood), or visual (e.g., coarse pattern).
Revised: The authors showed that the embedding dimensions of objects were interpretable and subjects could coherently label what the dimensions were “about,” ranging from categorical (e.g., animate, food) to functional (e.g., tool), structural (e.g., made of metal or wood), or visual (e.g., coarse pattern).

Original: The authors hypothesizedthat interpretability arose from combining positivity and sparsity constraints so that no object wasrepresented by every dimension, and most dimensions were present for only a few objects.
Revised: The authors hypothesizedthat interpretability arose from combining positivity and sparsity constraints so that no object wasrepresented by every dimension, and most dimensions were present for only a few objects.

Original: In addition
Revised: In addition

Original: SPoSE could predict human judgements close to the estimated best attainable performance [45, 18].
Revised: SPoSE could predict human judgements close to the estimated best attainable performance [45, 18].

Original: However, SPoSE has several limitations despite its notable performance.
Revised: However, SPoSE has several limitations despite its notable performance.

Original: The first stems from theuse of an ℓ 1 sparsity penalty to promote interpretability.
Revised: The first stems from theuse of an ℓ 1 sparsity penalty to promote interpretability.

Original: In SPoSE, 6 to 11 dominant dimensions foran object account for most of the prediction performance.
Revised: In SPoSE, 6 to 11 dominant dimensions foran object account for most of the prediction performance.

Original: These dimensions are different betweenobjects.
Revised: These dimensions are different betweenobjects.

Original: A potential issue with enforcing SPoSE to have even fewer dimensions is that it may causeexcessive shrinkage of the dominant values [1].
Revised: A potential issue with enforcing SPoSE to have even fewer dimensions is that it may causeexcessive shrinkage of the dominant values [1].

Original: Second, when inspecting the distributions of valuesacross objects, most SPoSE dimensions do not reflect the exponential prior induced by the ℓ 1 penalty.
Revised: Second, when inspecting the distributions of valuesacross objects, most SPoSE dimensions do not reflect the exponential prior induced by the ℓ 1 penalty.

Original: Overcoming this prior may lead to suboptimal performance and inconsistent solutions, specifically inlow data regimes.
Revised: Overcoming this prior may lead to suboptimal performance and inconsistent solutions, specifically inlow data regimes.

Original: Third, SPoSE uses an ad hoc criterion for determining the dimensionality of thesolution via an arbitrary threshold on the density of each dimension.
Revised: Third, SPoSE uses an ad hoc criterion for determining the dimensionality of thesolution via an arbitrary threshold on the density of each dimension.

Original: Finally, SPoSE has no criterionfor determining convergence of its optimization process, nor does it provide any formal guaranteeson the sample size needed to learn a model of desired complexity.
Revised: Finally, SPoSE has no criterionfor determining convergence of its optimization process, nor does it provide any formal guaranteeson the sample size needed to learn a model of desired complexity.

Original: To overcome these limitations we introduce VICE, a variational inference (VI) method for embeddingobject concepts with interpretable, sparse, and positive dimensions.
Revised: To overcome these limitations we introduce VICE, a variational inference (VI) method for embeddingobject concepts with interpretable, sparse, and positive dimensions.

Original: We start by discussing relatedwork, then describe the triplet task and SPoSE before presenting theory and experimental results.
Revised: We start by discussing relatedwork, then describe the triplet task and SPoSE before presenting theory and experimental results.

Original: Contribution 1: VICE solves major limitations of SPoSE First, VICE encourages shrinkagewhile allowing for small entry values by using a spike-and-slab prior [5, 11, 14, 28, 35, 39, 43].
Revised: Contribution 1: VICE solves major limitations of SPoSE First, VICE encourages shrinkagewhile allowing for small entry values by using a spike-and-slab prior [5, 11, 14, 28, 35, 39, 43].

Original: Wedeem this more appropriate than an exponential prior, because importance – the value an objecttakes in a dimension – is different from relevance – whether the dimension is applicable to thatobject – and they can be controlled separately.
Revised: Wedeem this more appropriate than an exponential prior, because importance – the value an objecttakes in a dimension – is different from relevance – whether the dimension is applicable to thatobject – and they can be controlled separately.

Original: Second, we use VI with a unimodal posterior forrepresenting each object in a dimension which yields a mean value and an uncertainty estimate.
Revised: Second, we use VI with a unimodal posterior forrepresenting each object in a dimension which yields a mean value and an uncertainty estimate.

Original: While unimodality makes it possible to use the mean values as representative object embeddings, theuncertainty estimates allow us to use a statistical procedure to automatically select the dimensionsthat explain the data.
Revised: While unimodality makes it possible to use the mean values as representative object embeddings, theuncertainty estimates allow us to use a statistical procedure to automatically select the dimensionsthat explain the data.

Original: Third, we use this procedure to introduce a convergence criterion that reliablyidentifies representational stability , i.e., consistency in the number of selected dimensions.
Revised: Third, we use this procedure to introduce a convergence criterion that reliablyidentifies representational stability , i.e., consistency in the number of selected dimensions.

Original: Contribution 2: A PAC bound on the generalization of SPoSE and VICE models This bound canbe used retrospectively , to provide guarantees about the generalization performance of a convergedmodel.
Revised: Contribution 2: A PAC bound on the generalization of SPoSE and VICE models This bound canbe used retrospectively , to provide guarantees about the generalization performance of a convergedmodel.

Original: Furthermore it can be used prospectively to determine the sample size required to identify arepresentation, given the number of objects and a maximum possible number of dimensions.
Revised: Furthermore it can be used prospectively to determine the sample size required to identify arepresentation, given the number of objects and a maximum possible number of dimensions.

Original: Contribution 3: Extensive evaluation of model performance, across multiple datasets Wecompare VICE with SPoSE over three different datasets.
Revised: Contribution 3: Extensive evaluation of model performance, across multiple datasets Wecompare VICE with SPoSE over three different datasets.

Original: One of these datasets contains concreteobjects, another consists of adjectives, and the third is composed of food items.
Revised: One of these datasets contains concreteobjects, another consists of adjectives, and the third is composed of food items.

Original: Experimentally wefind that VICE meets or beats the performance of SPoSE in modeling human behavior.
Revised: Experimentally wefind that VICE meets or beats the performance of SPoSE in modeling human behavior.

Original: Moreover,we find that VICE performs substantially better regarding the reproducibility of its dimensions anddisplays a lower variance for the number of (selected) dimensions.
Revised: Moreover,we find that VICE performs substantially better regarding the reproducibility of its dimensions anddisplays a lower variance for the number of (selected) dimensions.

Original: Both of these measures areparticularly important in the cognitive sciences.
Revised: Both of these measures areparticularly important in the cognitive sciences.

Original: Lastly, we compare VICE with SPoSE for reducedamounts of data and show that VICE has significantly better performance on all measures.
Revised: Lastly, we compare VICE with SPoSE for reducedamounts of data and show that VICE has significantly better performance on all measures.

Original: A PyTorch implementation of VICE with Continuous Integration (CI) is available on GitHub ( https://anonymous.4open.science/r/VICE-59F0 ) and will be de-anonymized upon acceptance.
Revised: A PyTorch implementation of VICE with Continuous Integration (CI) is available on GitHub ( https://anonymous.4open.science/r/VICE-59F0 ) and will be de-anonymized upon acceptance.

Original:  Related work
Revised:  Related work

Original: Navarro & Griffiths [31] introduced a method for learning semantic concept embeddings from itemsimilarity data which infers the number of embedding dimensions using the Indian Buffet Process(IBP)
Revised: Navarro & Griffiths [31] introduced a method for learning semantic concept embeddings from itemsimilarity data which infers the number of embedding dimensions using the Indian Buffet Process(IBP)

Original: Their approach relies on continuous-valued similarity ratings rather than discrete forcedchoice behavior and is not directly applicable to our setting.
Revised: Their approach relies on continuous-valued similarity ratings rather than discrete forcedchoice behavior and is not directly applicable to our setting.

Original: It is also challenging to scale the IBP tothe number of dimensions and samples in our work [36].
Revised: It is also challenging to scale the IBP tothe number of dimensions and samples in our work [36].

Original: Roads & Love [38] introduced a method forlearning object embeddings from behavior in an 8-rank-2 task.
Revised: Roads & Love [38] introduced a method forlearning object embeddings from behavior in an 8-rank-2 task.

Original: Their method predicts behavior from the embeddings by using active sampling [15] to query subjects with the most informative stimuli,yielding an object similarity matrix.
Revised: Their method predicts behavior from the embeddings by using active sampling [15] to query subjects with the most informative stimuli,yielding an object similarity matrix.

Original: Interpretability of embedding dimensions was not considered.
Revised: Interpretability of embedding dimensions was not considered.

Original: Other works have developed interpretable concept representations from text corpora.
Revised: Other works have developed interpretable concept representations from text corpora.

Original: Early methodsused word embeddings with positivity and sparsity constraints [30].
Revised: Early methodsused word embeddings with positivity and sparsity constraints [30].

Original: Later works in this directionused topic model representations of Wikipedia articles about objects [34], transformations of wordembeddings into sparse non-negative representations [41, 32], or predictions of properties [9] ordimensions [44].
Revised: Later works in this directionused topic model representations of Wikipedia articles about objects [34], transformations of wordembeddings into sparse non-negative representations [41, 32], or predictions of properties [9] ordimensions [44].

Original: Others have considered using text corpora in conjunction with imaging data [12, 7].
Revised: Others have considered using text corpora in conjunction with imaging data [12, 7].

Original: Finally, Derby et al.
Revised: Finally, Derby et al.

Original: [8] introduced a neural network function that maps the sparse feature spaceof a semantic property norm to the dense space of a word embedding, identifying informativecombinations of properties, and ranking candidate properties for new words.
Revised: [8] introduced a neural network function that maps the sparse feature spaceof a semantic property norm to the dense space of a word embedding, identifying informativecombinations of properties, and ranking candidate properties for new words.

Original:  Methods3.1 Triplet task
Revised:  Methods3.1 Triplet task

Original: The triplet task , also known as the odd-one-out triplet task, is used for discovering object concept embeddings from similarity judgments over a set of m different objects.
Revised: The triplet task , also known as the odd-one-out triplet task, is used for discovering object concept embeddings from similarity judgments over a set of m different objects.

Original: These judgmentsare collected from human participants who are given queries which consist of a triplet of objects (e.g., { “suit” , “flamingo” , “car” } ).
Revised: These judgmentsare collected from human participants who are given queries which consist of a triplet of objects (e.g., { “suit” , “flamingo” , “car” } ).

Original: Participants are asked to consider the three pairs in atriplet {{ “suit”,“flamingo” } , { “suit”,“car” } , { “flamingo”,“car” ) } , and to decide which pair is themost similar, leaving the third as the odd-one-out.
Revised: Participants are asked to consider the three pairs in atriplet {{ “suit”,“flamingo” } , { “suit”,“car” } , { “flamingo”,“car” ) } , and to decide which pair is themost similar, leaving the third as the odd-one-out.

Original: We assign each object a numerical index,e.g., 1 ← “aardvark” , . . .
Revised: We assign each object a numerical index,e.g., 1 ← “aardvark” , . . .

Original: , 1854 ← “zucchini”.
Revised: , 1854 ← “zucchini”.

Original: Let { y, z } denote the indices in this pair, e.g.,{ y, z } = { 268 , 609 } for “suit” and “flamingo.”
Revised: Let { y, z } denote the indices in this pair, e.g.,{ y, z } = { 268 , 609 } for “suit” and “flamingo.”

Original: A dataset D is a set of n ordered pairs of presentedtriplets and selected pairs.
Revised: A dataset D is a set of n ordered pairs of presentedtriplets and selected pairs.

Original: That is, D := (cid:0) { i s , j s , k s } , { y s , z s } (cid:1) n s =1 , where { y s , z s } ⊆ { i s , j s , k s } .
Revised: That is, D := (cid:0) { i s , j s , k s } , { y s , z s } (cid:1) n s =1 , where { y s , z s } ⊂ { i s , j s , k s } .

Original: ( c ) J ande r k( c ) C ha r l e s
Revised: ( c ) J ande r k( c ) C ha r l e s

Original: S ha r p ( c ) W a ll y
Revised: S ha r p ( c ) W a ll y

Original: G obe t z
Revised: G obe t z

Original: [Figure]
Revised: [Figure]

Original: Sparse Positive object Similarity Embedding (SPoSE) [45] is an approach for finding interpretableembedding dimensions from the triplet task.
Revised: Sparse Positive object Similarity Embedding (SPoSE) [45] is an approach for finding interpretableembedding dimensions from the triplet task.

Original: It does so by finding an embedding vector x i
Revised: It does so by finding an embedding vector x i

Original: =[ x
Revised: =[ x

Original: i 1 , . .
Revised: i 1 , . .

Original: , x id ] for every object i .
Revised: , x id ] for every object i .

Original: Let X denote
Revised: Let X denote

Original: the m × d matrix ( x 1 , . .
Revised: the m × d matrix ( x 1 , . .

Original: x m ) and S :
Revised: x m ) and S :

Original: = XX T be thesimilarity matrix, where S ij denote its entry at i, j .
Revised: = XX T be thesimilarity matrix, where S ij denote its entry at i, j .

Original: The probability of choosing { y s , z s } as the mostsimilar pair of objects, given an object triplet { i s , j s , k s } and the embedding matrix X , is modeled as
Revised: The probability of choosing { y s , z s } as the mostsimilar pair of objects, given an object triplet { i s , j s , k s } and the embedding matrix X , is modeled as

Original: [Equation]
Revised: [Equation]

Original: At this point we remark that each triplet { i s , j s , k s } is sampled uniformly at random from thecollection of all possible sets of object triplets T .
Revised: At this point we remark that each triplet { i s , j s , k s } is sampled uniformly at random from thecollection of all possible sets of object triplets T .

Original: That is, { i 1 , j 1 , k 1 } , . .
Revised: That is, { i 1 , j 1 , k 1 } , . .

Original: , { i n , j n , k n } i.i.d.
Revised: , { i n , j n , k n } i.i.d.

Original: ∼ U ( T ) .
Revised: ∼ U ( T ) .

Original: This is precisely stated in A.1.
Revised: This is precisely stated in A.1.

Original: SPoSE uses maximum a posteriori (MAP) estimation with a Laplaceprior under a non-negativity constraint (equivalent to an exponential prior) to find the most likelyembedding X for the training data D .
Revised: SPoSE uses maximum a posteriori (MAP) estimation with a Laplaceprior under a non-negativity constraint (equivalent to an exponential prior) to find the most likelyembedding X for the training data D .

Original: This leads to the training objective
Revised: This leads to the training objective

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: d (cid:88)
Revised: d (cid:88)

Original: j
Revised: j

Original: = | X ij | , where log p ( D| X ) =
Revised: = | X ij | , where log p ( D| X ) =

Original: constant + (cid:80) ns =1 log p ( { y s , z s }|{ i s , j s , k s } , X ) (see A.1 for derivation) and λis determined using cross-validation.
Revised: constant + (cid:80) ns =1 log p ( { y s , z s }|{ i s , j s , k s } , X ) (see A.1 for derivation) and λis determined using cross-validation.

Original: 3.3 VICE
Revised: 3.3 VICE

Original: In contrast to SPoSE, we use VI [4] instead of a MAP estimate for approximating the posteriorprobability p ( X |D ) .
Revised: In contrast to SPoSE, we use VI [4] instead of a MAP estimate for approximating the posteriorprobability p ( X |D ) .

Original: In VICE we impose additional constraints on the embedding matrix X by usinga prior that encourages shrinkage while allowing entries in X to be close to zero (approx. sparse).
Revised: In VICE we impose additional constraints on the embedding matrix X by usinga prior that encourages shrinkage while allowing entries in X to be close to zero (approx. sparse).

Original: 3.3.1 Variational Bayesian Inference
Revised: 3.3.1 Variational Bayesian Inference

Original: For VICE we consider approximating p ( X |D ) with a variational distribution, q θ ( X ) , where q θ ∈ Q ,and θ is optimized to minimize the KL divergence to the true posterior, p ( X |D ) .
Revised: For VICE we consider approximating p ( X |D ) with a variational distribution, q θ ( X ) , where q θ ∈ Q ,and θ is optimized to minimize the KL divergence to the true posterior, p ( X |D ) .

Original: In VICE the KLdivergence objective function (derived in A.2) is
Revised: In VICE the KLdivergence objective function (derived in A.2) is

Original: [Equation]
Revised: [Equation]

Original: Variational distribution VI requires a choice of a parametric variational distribution q ∈ Q .
Revised: Variational distribution VI requires a choice of a parametric variational distribution q ∈ Q .

Original: For
Revised: For

Original: VICE we use a Gaussian distribution with a diagonal covariance matrix q θ ( X ) =
Revised: VICE we use a Gaussian distribution with a diagonal covariance matrix q θ ( X ) =

Original: N ( µ, diag ( σ 2 ))where θ = { µ, σ } .
Revised: N ( µ, diag ( σ 2 ))where θ = { µ, σ } .

Original: Therefore, each embedding dimension has a mean and a standard deviation .
Revised: Therefore, each embedding dimension has a mean and a standard deviation .

Original: Wedeem a Gaussian variational distribution appropriate for a variety of reasons.
Revised: Wedeem a Gaussian variational distribution appropriate for a variety of reasons.

Original: First, under certainconditions, the posterior is Gaussian in the infinite-data limit [23].
Revised: First, under certainconditions, the posterior is Gaussian in the infinite-data limit [23].

Original: Second, a unimodal variationaldistribution makes it possible to use µ as representative object embeddings.
Revised: Second, a unimodal variationaldistribution makes it possible to use µ as representative object embeddings.

Original: A fixed representations isuseful for downstream use cases that rely on a single embedding vector for each object.
Revised: A fixed representations isuseful for downstream use cases that rely on a single embedding vector for each object.

Original: Use casesrange from using the embeddings as targets in a regression task over unsupervised clustering of theembeddings to interpreting the embeddings.
Revised: Use casesrange from using the embeddings as targets in a regression task over unsupervised clustering of theembeddings to interpreting the embeddings.

Original: More complex variational families are not as practicalfor this.
Revised: More complex variational families are not as practicalfor this.

Original: Third, a Gaussian posterior is a computationally convenient choice.
Revised: Third, a Gaussian posterior is a computationally convenient choice.

Original: Similarly to Titsias & Lázaro-Gredilla [42], we use a Monte Carlo (MC) approximation of (2) bysampling a limited number of X s from q θ ( X ) during training.
Revised: Similarly to Titsias & Lázaro-Gredilla [42], we use a Monte Carlo (MC) approximation of (2) bysampling a limited number of X s from q θ ( X ) during training.

Original: We generate X with the reparameterization trick [22, 37], X θ,ϵ = µ + σ ⊙ ϵ , where ϵ ∈ R m × d is entrywise N (0 , 1) , and ⊙ denotes the
Revised: We generate X with the reparameterization trick [22, 37], X θ,ϵ = µ + σ ⊙ ϵ , where ϵ ∈ R m × d is entrywise N (0 , 1) , and ⊙ denotes the

Original: Hadamard (element-wise) product.
Revised: Hadamard (element-wise) product.

Original: This leads to the objectivearg
Revised: This leads to the objectivearg

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: We apply a ReLU function, denoted by [ · ] + , to the sampled X θ,ϵ values to guarantee that X θ,ϵ ∈ 141 R m × d + .
Revised: We apply a ReLU function, denoted by [ · ] + , to the sampled X θ,ϵ values to guarantee that X θ,ϵ ∈ 141 R m × d + .

Original: As commonly done in the Dropout and Bayesian Neural Network literature [40, 5, 13, 26],we set R = 1 for computational efficiency during the optimization process.
Revised: As commonly done in the Dropout and Bayesian Neural Network literature [40, 5, 13, 26],we set R = 1 for computational efficiency during the optimization process.

Original: Posterior probability estimation Computational efficiency at inference time is not as critical as itis during training.
Revised: Posterior probability estimation Computational efficiency at inference time is not as critical as itis during training.

Original: Therefore, we can get a better estimate of the posterior probability distributionover the three possible odd one-one-out choices by letting R ≫ 1 .
Revised: Therefore, we can get a better estimate of the posterior probability distributionover the three possible odd one-one-out choices by letting R ≫ 1 .

Original: Using the optimized variationalposterior, q ˆ θ ( X ) , we approximate the probability distribution with an MC estimate [16, 5, 26, 4] from
Revised: Using the optimized variationalposterior, q ˆ θ ( X ) , we approximate the probability distribution with an MC estimate [16, 5, 26, 4] from

Original: R samples X ( r ) =
Revised: R samples X ( r ) =

Original: X ˆ θ,ϵ ( r ) for r = 1 , . . .
Revised: X ˆ θ,ϵ ( r ) for r = 1 , . . .

Original: , R , yielding
Revised: , R , yielding

Original: [Equation]
Revised: [Equation]

Original: Spike-and-Slab prior As discussed above, SPoSE induces sparsity through an ℓ 1 penalty which,along with the non-negativity constraint, is equivalent to using an exponential prior.
Revised: Spike-and-Slab prior As discussed above, SPoSE induces sparsity through an ℓ 1 penalty which,along with the non-negativity constraint, is equivalent to using an exponential prior.

Original: Throughexamination of the publicly available histograms of weight values in the two most important SPoSEdimensions (see Figure 2 in A.4), we observed that the dimensions did not resemble an exponentialdistribution.
Revised: Throughexamination of the publicly available histograms of weight values in the two most important SPoSEdimensions (see Figure 2 in B.3), we observed that the dimensions did not resemble an exponentialdistribution.

Original: Instead, they contained a spike of probability at zero and a wide slab of probability forthe non-zero values.
Revised: Instead, they contained a spike of probability at zero and a wide slab of probability forthe non-zero values.

Original: To model this, we use a spike-and-slab Gaussian mixture prior [5, 11, 14, 20, 25],
Revised: To model this, we use a spike-and-slab Gaussian mixture prior [5, 11, 14, 20, 25],

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which encourages shrinkage
Revised: which encourages shrinkage

Original: This prior has three parameters, σ spike , σ slab , and π spike .
Revised: This prior has three parameters, σ spike , σ slab , and π spike .

Original: π spike is theprobability that an embedding dimension is drawn from the spike Gaussian.
Revised: π spike is theprobability that an embedding dimension is drawn from the spike Gaussian.

Original: Since spike and slabdistributions are mathematically interchangeable, by convention we require that σ spike ≪ σ slab .
Revised: Since spike and slabdistributions are mathematically interchangeable, by convention we require that σ spike ≪ σ slab .

Original: 3.3.2 Dimensionality reduction and convergence
Revised: 3.3.2 Dimensionality reduction and convergence

Original: For interpretability purposes it is desirable for the object embedding dimensionality, d , to be as smallas possible.
Revised: For interpretability purposes it is desirable for the object embedding dimensionality, d , to be as smallas possible.

Original: In contrast to SPoSE, which employs a user-defined threshold to prune dimensions, VICEexploits the uncertainty estimates for embedding values to select a subset of informative dimensions.
Revised: In contrast to SPoSE, which employs a user-defined threshold to prune dimensions, VICEexploits the uncertainty estimates for embedding values to select a subset of informative dimensions.

Original: The pruning procedure works by assigning importance scores to each of the d dimensions, whichreflect the number of objects that we can confidently say have a non-zero weight in a dimension.
Revised: The pruning procedure works by assigning importance scores to each of the d dimensions, whichreflect the number of objects that we can confidently say have a non-zero weight in a dimension.

Original: Tocompute the score, we use the variational embedding for each object i and dimension j – locationµ ij and scale σ ij parameters – to compute the posterior probability that the weight is truncated tozero according to the left tail of a Gaussian distribution with that location and scale (see §3.3.1).
Revised: Tocompute the score, we use the variational embedding for each object i and dimension j – locationµ ij and scale σ ij parameters – to compute the posterior probability that the weight is truncated tozero according to the left tail of a Gaussian distribution with that location and scale (see §3.3.1).

Original: Thisgives us a posterior probability of the weight being zero for each object within a dimension [16].
Revised: Thisgives us a posterior probability of the weight being zero for each object within a dimension [16].

Original: Tocalculate the overall importance of a dimension, we estimate the number of objects that have non-zeroweights, while controlling False Discovery Rate (FDR) [2] with α = 0 .
Revised: Tocalculate the overall importance of a dimension, we estimate the number of objects that have non-zeroweights, while controlling False Discovery Rate (FDR) [2] with α = 0 .

Original: We define importance ofeach dimension j to be the number of objects for which P ( X ij > 0) ≥ .
Revised: We define importance ofeach dimension j to be the number of objects for which P ( X ij > 0) ≥ .

Original: 95 holds.
Revised: 95 holds.

Original: After convergence,we prune the model by removing dimensions with 5 or fewer statistically significant objects.
Revised: After convergence,we prune the model by removing dimensions with 5 or fewer statistically significant objects.

Original: This is acommonly used reliability threshold in semantic property norms (e.g., [27, 9]).
Revised: This is acommonly used reliability threshold in semantic property norms (e.g., [27, 9]).

Original: In gradient-based optimization, the gradient of an objective function with respect to the parametersof a model, ∇L ( θ ) , is used to iteratively find parameters ˆ θ that minimize that function.
Revised: In gradient-based optimization, the gradient of an objective function with respect to the parametersof a model, ∇L ( θ ) , is used to iteratively find parameters ˆ θ that minimize that function.

Original: We usea representational stability criterion to determine convergence.
Revised: We usea representational stability criterion to determine convergence.

Original: That is, the optimization processhalts when the number of identified dimensions - as described above - has not changed by a singledimension over the past L epochs (e.g., L = 500 ).
Revised: That is, the optimization processhalts when the number of identified dimensions - as described above - has not changed by a singledimension over the past L epochs (e.g., L = 500 ).

Original: Given that our goal is to find stable estimates ofthe number of dimensions, we considered this to be more appropriate than other convergence criteriasuch as evaluating the cross-entropy error on a validation set or evidence-based criteria [24, 10].
Revised: Given that our goal is to find stable estimates ofthe number of dimensions, we considered this to be more appropriate than other convergence criteriasuch as evaluating the cross-entropy error on a validation set or evidence-based criteria [24, 10].

Original: Forfurther details on convergence and the optimization process see A.3.
Revised: Forfurther details on convergence and the optimization process see B.1.

Original: 3.4 Sample complexity bound
Revised: 3.4 Sample complexity bound

Original: We use statistical learning theory to obtain estimates of the sample size needed to appropriately constrain VICE (and SPoSE) models.
Revised: We use statistical learning theory to obtain estimates of the sample size needed to appropriately constrain VICE (and SPoSE) models.

Original: These estimates can be used retrospectively , to obtain probabilisticguarantees on the generalization of a trained model to unseen data, or prospectively , to decide howmuch data to collect for a study.
Revised: These estimates can be used retrospectively , to obtain probabilisticguarantees on the generalization of a trained model to unseen data, or prospectively , to decide howmuch data to collect for a study.

Original: For VICE, and in this analysis only, we extract µ and use it as a fixedembedding to predict the most likely odd-one-out for a query triplet rather than sampling embeddingsfrom the variational distribution.
Revised: For VICE, and in this analysis only, we extract µ and use it as a fixedembedding to predict the most likely odd-one-out for a query triplet rather than sampling embeddingsfrom the variational distribution.

Original: To obtain a useful bound we make two assumptions.
Revised: To obtain a useful bound we make two assumptions.

Original: First, we assume that there exists an upperbound M on the largest value in the embedding.
Revised: First, we assume that there exists an upperbound M on the largest value in the embedding.

Original: Second, we assume that the embeddings obtainedby either SPoSE or VICE can be quantized in a relatively coarse fashion with marginal losses inpredictive performance.
Revised: Second, we assume that the embeddings obtainedby either SPoSE or VICE can be quantized in a relatively coarse fashion with marginal losses inpredictive performance.

Original: We can choose a certain discretization scale ∆ , e.g., ∆ = 0 . 5 , and roundembedding values to a nonnegative integer multiple of ∆ .
Revised: We can choose a certain discretization scale ∆ , e.g., ∆ = 0 . 5 , and roundembedding values to a nonnegative integer multiple of ∆ .

Original: While we employ this quantization mainlyto use learning theory bounds for finite hypothesis classes, it could have benefits for interpretationas well (e.g., a dimension could consist of labels such as zero (0), very low (0.5), low (1), medium(1.5), high (2)).
Revised: While we employ this quantization mainlyto use learning theory bounds for finite hypothesis classes, it could have benefits for interpretationas well (e.g., a dimension could consist of labels such as zero (0), very low (0.5), low (1), medium(1.5), high (2)).

Original: In all datasets we have used, SPoSE and VICE embeddings with reasonablepriors had values below 2 .
Revised: In all datasets we have used, SPoSE and VICE embeddings with reasonablepriors had values below 2 .

Original: 7 (see A.7).
Revised: 7 (see E.2).

Original: Given an upper bound M and a discretization scale ∆one can consider all possible ways of populating the matrix X with weights limited to the set
Revised: Given an upper bound M and a discretization scale ∆one can consider all possible ways of populating the matrix X with weights limited to the set

Original: S = { 0 , ∆ , . . .
Revised: S = { 0 , ∆ , . . .

Original: , ( k − 1)∆ , k ∆ } .
Revised: , ( k − 1)∆ , k ∆ } .

Original: The following bound tells us that with high probability the estimatederror rate and true error rate for such discretized X matrices are close to each other.
Revised: The following bound tells us that with high probability the estimatederror rate and true error rate for such discretized X matrices are close to each other.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Given δ > 0 , ϵ > 0 , and n ≥c 2 / (cid:0) 2 ϵ 2 (cid:1) ( md log( k + 1) + log (2 /δ )) trainingtriplets ( { y s , z s } , { i s , j s , k 1 s } ) ns =1 , then
Revised: Given δ > 0 , ϵ > 0 , and n ≥ ( md log( k + 1) + log (1 /δ )) / (cid:0) 2 ϵ 2 (cid:1) trainingsamples ( { y s , z s } , { i s , j s , k 1 s } ) ns =1 , then

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof sketch.

Original: This bound follows from applying Hoeffding’s inequality to the ( k + 1) m × d elements of S m × d combined with a union bound.
Revised: This bound follows from applying Hoeffding’s inequality to the ( k + 1) m × d elementsof S m × d combined with a union bound.

Original: This proof is virtually identical to that of Theorem 2.13 in
Revised: This proof is virtually identical to that of Theorem 2.13 in

Original: Mohri et al.
Revised: Mohri et al.

Revised: A complete proof can be found in C.

Original: To use the bound we first decide on the number of quantization steps k = ⌈ M/ ∆ ⌉ .
Revised: To use the bound we first decide on the number of quantization steps k = ⌈ M/ ∆ ⌉ .

Original: The probabilityof violating the bound δ is analogous to the Type I error control in hypothesis testing, often 0 .
Revised: The probabilityof violating the bound δ is analogous to the Type I error control in hypothesis testing, often 0 .

Original: As the bound depends very weakly on δ , it is convenient to use δ = 0 .
Revised: As the bound depends very weakly on δ , it is convenient to use δ = 0 .

Original: The error toleranceϵ does have a major effect on the sample size estimate provided by the bound.
Revised: The error toleranceϵ does have a major effect on the sample size estimate provided by the bound.

Original: For the bound tobe practically useful, ϵ has to be smaller than the difference between the training error – averagezero-one loss over the training set examples – and random guessing error.
Revised: For the bound tobe practically useful, ϵ has to be smaller than the difference between the training error – averagezero-one loss over the training set examples – and random guessing error.

Original: For the datasets we use inthis paper we observed a difference of 0 .
Revised: For the datasets we use inthis paper we observed a difference of 0 .

Original: 2 − 0 .
Revised: 2 − 0 .

Original: A conservative choice would halve the accuracygap of 0.2, giving ϵ = 0 . 1 .
Revised: A conservative choice would halve the accuracygap of 0.2, giving ϵ = 0 . 1 .

Original: Together, these values result in a prospective sample size estimate ofn ≈ 50 · ( md log(5) + log(2000)) .
Revised: Together, these values result in a prospective sample size estimate ofn ≈ 50 · ( md log(5) + log(1000)) .

Original: To use the bound retrospectively, we fix n , δ while varying ∆ toget a guarantee on ϵ , which in turn yields a probabilistic upper bound on the generalization error foran embedding, R ( X ) ≤ ˆ R ( X ) + ϵ .
Revised: To use the bound retrospectively, we fix n , δ while varying ∆ toget a guarantee on ϵ , which in turn yields a probabilistic upper bound on the generalization error foran embedding, R ( X ) ≤ ˆ R ( X ) + ϵ .

Original: For more details, see Algorithm 1 in A.6.
Revised: For more details, see Algorithm 1 in E.1.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Data
Revised: 4.1 Data

Original: Our experiments were performed on three datasets: T HINGS [45, 18], A DJECTIVES , and F OOD .
Revised: Our experiments were performed on three datasets: T HINGS [45, 18], A DJECTIVES , and F OOD .

Original: T HINGS and A DJECTIVES contain random samples from all possible triplet combinations for 1,854objects and 2,372 adjectives, respectively.
Revised: T HINGS and A DJECTIVES contain random samples from all possible triplet combinations for 1,854objects and 2,372 adjectives, respectively.

Original: T HINGS and A DJECTIVES are each comprised of a largetraining dataset which contains no “repeats,” i.e., there is only one human response for each triplet( ∼ 1 .
Revised: T HINGS and A DJECTIVES are each comprised of a largetraining dataset which contains no “repeats,” i.e., there is only one human response for each triplet( ∼ 1 .

Original: 5 M triplets for T HINGS and ∼ 800 K triplets for A DJECTIVES ).
Revised: 5 M triplets for T HINGS and ∼ 800 K triplets for A DJECTIVES ).

Original: For each of these trainingdatasets 10% of the triplets are assigned to a predefined validation set.
Revised: For each of these trainingdatasets 10% of the triplets are assigned to a predefined validation set.

Original: The test sets for both containthe results for 1,000 random triplets that are not contained in the training data, with 25 repeats foreach triplet.
Revised: The test sets for both containthe results for 1,000 random triplets that are not contained in the training data, with 25 repeats foreach triplet.

Original: F OOD contains data for every possible triplet combination for 36 objects and repeats forsome triplets.
Revised: F OOD contains data for every possible triplet combination for 36 objects and repeats forsome triplets.

Original: We partitioned this dataset into train (45%), validation (5%), and test (50%) sets withdisjoint triplets.
Revised: We partitioned this dataset into train (45%), validation (5%), and test (50%) sets withdisjoint triplets.

Original: A.8 provides details about stimuli and data collection, and quality control.
Revised: F provides details about stimuli, data collection, and quality control.

Original: 4.2 Experimental setup
Revised: 4.2 Experimental setup

Original: We implemented both SPoSE and VICE in PyTorch [33] using Adam [21] with η = 0 . 001 .
Revised: We implemented both SPoSE and VICE in PyTorch [33] using Adam [21] with η = 0 . 001 .

Original: To findthe optimal VICE hyperparameter combination we performed a grid search over σ spike , σ slab , π spike ,optimizing (3) and evaluating each model on the validation set.
Revised: To findthe optimal VICE hyperparameter combination we performed a grid search over σ spike , σ slab , π spike ,optimizing (3) and evaluating each model on the validation set.

Original: All experiments were performedover 20 different random initializations.
Revised: All experiments were performedover 20 different random initializations.

Original: For VICE we observed that the hyperparameter values withthe lowest average cross-entropy error on the validation set were highly similar across the differentdatasets (see Table 1 in A.5).
Revised: For VICE we observed that the hyperparameter values withthe lowest average cross-entropy error on the validation set were highly similar across the differentdatasets (see Table 1 in D).

Original: For more information about the experimental setup, including detailsabout training, weight initialization(s), hyperparameter grid, and optimal combination, see A.5.
Revised: For more information about the experimental setup, including detailsabout training, weight initialization(s), hyperparameter grid, and optimal combination, see D.

Original: 4.3 Evaluation measures
Revised: 4.3 Evaluation measures

Original: VICE estimates p ( { y, z }|{ i, j, k } ) , the model’s softmax probability distribution over a given triplet(see (4)).
Revised: VICE estimates p ( { y, z }|{ i, j, k } ) , the model’s softmax probability distribution over a given triplet(see (4)).

Original: The first evaluation measure we consider is the prediction accuracy with respect to the correct human choice { y, z } , where the predicted pair is arg max { y,z } ˆ p ( { y, z }|{ i, j, k } ) .
Revised: The first evaluation measure we consider is the prediction accuracy with respect to the correct human choice { y, z } , where the predicted pair is arg max { y,z } ˆ p ( { y, z }|{ i, j, k } ) .

Original: Weestimate an upper bound on the prediction accuracy by using the repeats in the test set.
Revised: Weestimate an upper bound on the prediction accuracy by using the repeats in the test set (see D).

Original: A Bayesoptimal classifier would predict the majority outcome for any triplet, whereas random guessing givesan accuracy of 1 / 3 .
Revised: A Bayes-optimal classifier would predict the majority outcome for any triplet, whereas random guessinggives an accuracy of 1 / 3 .

Original: Note that the triplet task is subjective and thus there is not necessarily a definitivecorrect answer for a given triplet.
Revised: Note that the triplet task is subjective and thus there is not necessarily adefinitive correct answer for a given triplet.

Original: If a test set contains multiple responses for a triplet, it providesinformation about the relative similarities of the three object pairs.
Revised: If a test set contains multiple responses for a triplet, itprovides information about the relative similarities of the three object pairs.

Original: This lets us calculate probabilitydistributions over answers for each triplet.
Revised: This lets us calculateprobability distributions over answers for each triplet.

Original: Approximating these distributions closely is important incognitive science applications.
Revised: Approximating these distributions closelyis important in cognitive science applications.

Original: Therefore, we additionally evaluate the models by computing the KLdivergences between the models’ posterior probability estimates ˆ p ( { y, z }|{ i, j, k } ) (see (4)) and thehuman probability distributions inferred from the test set across all test triplets and report the average.
Revised: Therefore, we additionally evaluate the models bycomputing the KL divergences between the models’ posterior probability estimates ˆ p ( { y, z }|{ i, j, k } )(see (4)) and the human probability distributions inferred from the test set across all test triplets andreport the average.

Original: 4.4 Results on THINGS
Revised: 4.4 Results on THINGS

Original: Full dataset For this experiment we compared a representative model from VICE and SPoSE.
Revised: Full dataset For this experiment we compared a representative model from VICE and SPoSE.

Original: Therepresentative models were chosen to be those with the median cross-entropy error on the validationset over the 20 seeds.
Revised: Therepresentative models were chosen to be those with the median cross-entropy error on the validationset over the 20 seeds.

Original: For VICE, we set the number of MC samples to R = 50 .
Revised: For VICE, we set the number of MC samples to R = 50 .

Original: On the test set, VICEand SPoSE achieved similar prediction accuracies of 0 . 638 and 0 .
Revised: On the test set, VICEand SPoSE achieved similar prediction accuracies of 0 . 638 and 0 .

Original: 637 respectively (estimated upperbound for accuracy was 0 . 673 ).
Revised: 637 respectively (estimated upperbound for accuracy was 0 . 673 ).

Original: Likewise, VICE and SPoSE achieved similar average KL divergencesof 0 . 100 and 0 .
Revised: Likewise, VICE and SPoSE achieved similar average KL divergencesof 0 . 100 and 0 .

Original: 103 respectively (random guessing KL divergence was 0 . 366 ).
Revised: 103 respectively (random guessing KL divergence was 0 . 366 ).

Original: The differencesbetween the median model accuracies and KL divergences were not statistically significant accordingto a two-sided paired t -test over individual test triplets.
Revised: The differencesbetween the median model accuracies and KL divergences were not statistically significant accordingto a two-sided paired t -test over individual test triplets.

Original: Hence, VICE and SPoSE predicted tripletsapproximately equally well when trained on the full dataset.
Revised: Hence, VICE and SPoSE predicted tripletsapproximately equally well when trained on the full dataset.

Original: This is not surprising, however, since
Revised: This is not surprising, however, since

Original: Bayesian methods based on MC sampling become more like deterministic maximum likelihoodestimation in the infinite data limit.
Revised: Bayesian methods based on MC sampling become more like deterministic maximum likelihoodestimation in the infinite data limit.

Original: If n → ∞ , then the left-hand side in the expectation of (2), whichaccounts for the contribution of the prior, goes to zero.
Revised: If n → ∞ , then the left-hand side in the expectation of (2), whichaccounts for the contribution of the prior, goes to zero.

Original: Conversely, the effects of the prior are moreprominent when n is small, as we show in the following section.
Revised: Conversely, the effects of the prior are moreprominent when n is small, as we show in the following section.

Original: Data efficiency experiments Performance on small datasets is important in cognitive science, sincebehavioral experiments often have low sample sizes (e.g., tens to hundreds of volunteer in-labsubjects) compared to THINGS, which is particularly large, or they can be costly to crowdsource.
Revised: Data efficiency experiments Performance on small datasets is important in cognitive science, sincebehavioral experiments often have low sample sizes (e.g., tens to hundreds of volunteer in-labsubjects) compared to THINGS, which is particularly large, or they can be costly to crowdsource.

Original: To test whether VICE can model the data better than SPoSE in low sample regimes we performedexperiments training on smaller subsets of the training dataset.
Revised: To test whether VICE can model the data better than SPoSE in low sample regimes we performedexperiments training on smaller subsets of the training dataset.

Original: We did this for subsets with sizesequal to 5%, 10%, 20%, and 50% of the training dataset.
Revised: We did this for subsets with sizesequal to 5%, 10%, 20%, and 50% of the training dataset.

Original: For each size we divided the training setinto equal partitions and performed an experiment for each partition, thereby giving multiple resultsfor each subset size.
Revised: For each size we divided the training setinto equal partitions and performed an experiment for each partition, thereby giving multiple resultsfor each subset size.

Original: Validation and held-out test sets remained unchanged.
Revised: Validation and held-out test sets remained unchanged.

Original: In Figure 2 we show theaverage KL divergence (left) and prediction accuracy (middle) across data partitions and 20 randomseeds (used to initialize model parameters) for various dataset sizes.
Revised: In Figure 2 we show theaverage KL divergence (left) and prediction accuracy (middle) across data partitions and 20 randomseeds (used to initialize model parameters) for various dataset sizes.

Original: The average performance acrosspartitions was used to compute the confidence intervals (CIs).
Revised: The average performance acrosspartitions was used to compute the confidence intervals (CIs).

Original: The differences in prediction accuraciesand KL divergences between VICE and SPoSE were notable for the 5% and 10% data subsets.
Revised: The differences in prediction accuraciesand KL divergences between VICE and SPoSE were notable for the 5% and 10% data subsets.

Original: Inthe former, SPoSE predicted only slightly better than random guessing; in the latter, SPoSE showeda large variability between random seeds and data partitions, as can be seen in the 95% CIs.
Revised: Inthe former, SPoSE predicted only slightly better than random guessing; in the latter, SPoSE showed a large variability between random seeds and data partitions, as can be seen in the 95% CIs.

Original: In comparison VICE showed a small variation in the two performance metrics across random seedsand performed much better than random guessing.
Revised: Incomparison VICE showed a small variation in the two performance metrics across random seedsand performed much better than random guessing.

Original: The differences between VICE and SPoSE forthe 5% and 10% subset scenarios were statistically significant according to a two-sided paired t -testcomparing individual triplet predictions between median models ( p < 0 .
Revised: The differences between VICE and SPoSE forthe 5% and 10% subset scenarios were statistically significant according to a two-sided paired t -testcomparing individual triplet predictions between median models ( p < 0 .

Original: For the full trainingdataset VICE used significantly fewer dimensions than SPoSE to achieve the same performance.
Revised: For the full trainingdataset VICE used significantly fewer dimensions than SPoSE to achieve the same performance.

Original: 4.5 Other results
Revised: 4.5 Other results

Original: Other datasets On the A DJECTIVES test set, VICE and SPoSE had accuracies of 0 .
Revised: Other datasets On the A DJECTIVES test set, VICE and SPoSE had accuracies of 0 .

Original: 559 and 0 .
Revised: 559 and 0 .

Original: 562 ,respectively (estimated upper bound was 0 . 607 ), and KL divergences of 0 . 083 and 0 .
Revised: 562 ,respectively (estimated upper bound was 0 . 607 ), and KL divergences of 0 . 083 and 0 .

Original: 088 , respectively(random guessing was 0 . 366 ).
Revised: 088 , respectively(random guessing was 0 . 366 ).

Original: On the F OOD test set, VICE and SPoSE median models had accuraciesof 0 .
Revised: On the F OOD test set, VICE and SPoSE median models had accuraciesof 0 .

Original: 693 and 0 .
Revised: 693 and 0 .

Original: 698 , respectively.
Revised: 698 , respectively.

Original: The differences in accuracy (and KL divergence for A DJECTIVES )between VICE and SPoSE for both datasets were not statistically significant according to two-sidedpaired t -tests on the median model predictions across triplets on the test set.
Revised: The differences in accuracy (and KL divergence for A DJECTIVES )between VICE and SPoSE for both datasets were not statistically significant according to two-sidedpaired t -tests on the median model predictions across triplets on the test set.

Original: However, for F OOD ,
Revised: However, for F OOD ,

Original: VICE used significantly fewer dimensions than SPoSE to achieve similar performance (see Table 1).
Revised: VICE used significantly fewer dimensions than SPoSE to achieve similar performance (see Table 1).

Original: Hyperparameters We observed that VICE’s performance is fairly insensitive to hyperparameterselection.
Revised: Hyperparameters We observed that VICE’s performance is fairly insensitive to hyperparameterselection.

Original: Using default hyperparameters, σ spike = 0 .
Revised: Using default hyperparameters, σ spike = 0 .

Original: 25 , σ slab = 1 . 0 , π sigma = 0 .
Revised: 25 , σ slab = 1 . 0 , π sigma = 0 .

Original: 5 , yields an accuracyscore within 0 .
Revised: 5 , yields an accuracyscore within 0 .

Original: 015 of the best cross-validated model in the full dataset setting, on all three datasets.
Revised: 015 of the best cross-validated model in the full dataset setting, on all three datasets.

Original: 4.6 Reproducibility and Representational stability
Revised: 4.6 Reproducibility and Representational stability

Original: Reproducibility As mentioned in § 1, a key criterion for learning concept representations beyondpredictive performance is reproducibility , i.e., learning similar representations for different randominitializations on the same training data.
Revised: Reproducibility As mentioned in § 1, a key criterion for learning concept representations beyondpredictive performance is reproducibility , i.e., learning similar representations for different randominitializations on the same training data.

Original: To evaluate this we compared the learned embeddings from20 differently initialized VICE and SPoSE models.
Revised: To evaluate this we compared the learned embeddings from20 differently initialized VICE and SPoSE models.

Original: The first aspect of reproducibility we investigateis whether the models yield a consistent number of embedding dimensions across random seeds.
Revised: The first aspect of reproducibility we investigateis whether the models yield a consistent number of embedding dimensions across random seeds.

Original: As reported in Table 1, VICE yields fewer dimensions than SPoSE with less variance in the numberof dimensions for all three datasets.
Revised: As reported in Table 1, VICE yields fewer dimensions than SPoSE with less variance in the numberof dimensions for all three datasets.

Original: The embedding dimensionality was statistically significantlysmaller according to an independent t -test for T HINGS ( p < 0 .
Revised: The embedding dimensionality was statistically significantlysmaller according to an independent t -test for T HINGS ( p < 0 .

Original: 001 ) and F OOD ( p < 0 . 001 ), butnot for A DJECTIVES ( p = 0 . 108 ).
Revised: 001 ) and F OOD ( p < 0 . 001 ), butnot for A DJECTIVES ( p = 0 . 108 ).

Original: The difference in the standard deviations for A DJECTIVESwas statistically significant according to a two-sided F -test ( p = 0 . 002 ), but was not statisticallysignificant for T HINGS ( p = 0 . 283 ) or F OOD ( p = 0 . 378 ).
Revised: The difference in the standard deviations for A DJECTIVESwas statistically significant according to a two-sided F -test ( p = 0 . 002 ), but was not statisticallysignificant for T HINGS ( p = 0 . 283 ) or F OOD ( p = 0 . 378 ).

Revised: In addition, we observed that the identifieddimensionality is consistent as long as d is chosen to be sufficiently large (see D).

Original: [Table]
Revised: [Table]

Original: The second aspect to reproducibility we examine is the degree to which the identified dimensions aresimilar across random initializations up to a permutation.
Revised: The second aspect to reproducibility we examine is the degree to which the identified dimensions aresimilar across random initializations up to a permutation.

Original: To calculate the number of reproducibledimensions we associate each embedding dimension of a model with the most similar embeddingdimension across the other models.
Revised: To calculate the number of reproducibledimensions we associate each embedding dimension of a model with the most similar embeddingdimension across the other models.

Original: We quantify reproducibility of a dimension as the average Pearsoncorrelation between one dimension and its best match across the 19 remaining models.
Revised: We quantify reproducibility of a dimension as the average Pearsoncorrelation between one dimension and its best match across the 19 remaining models.

Original: In Table 1we report the average relative number of dimensions with a correlation > 0 . 8 across models.
Revised: In Table 1we report the average relative number of dimensions with a correlation > 0 . 8 across models.

Original: Theembedding dimensions in VICE were more reproducible than those in SPoSE.
Revised: Theembedding dimensions in VICE were more reproducible than those in SPoSE.

Original: The difference inaverage reproducibility scores was statistically significant according to a two-sided, independentt -test for F OOD ( p = 0 . 030 ) and T HINGS ( p = 0 . 008 ), but not for A DJECTIVES ( p = 0 . 466 ).
Revised: The difference inaverage reproducibility scores was statistically significant according to a two-sided, independentt -test for F OOD ( p = 0 . 030 ) and T HINGS ( p = 0 . 008 ), but not for A DJECTIVES ( p = 0 . 466 ).

Original: Representational stability For all three datasets VICE found reproducible and stable dimensionsacross all 20 random initializations and converged when the embedding dimensionality had not changed over the past L = 500 epochs.
Revised: Representational stability For all three datasets VICE found reproducible and stable dimensionsacross all 20 random initializations and converged when the embedding dimensionality had notchanged over the past L = 500 epochs.

Original: The median number of epochs to achieve representationalstability was comparable for the similarly-sized T HINGS and A DJECTIVES datasets, but occurredlater for F OOD (likely due to the smaller number of gradient updates per epoch).
Revised: The median number of epochs to achieve representationalstability was comparable for the similarly-sized T HINGS and A DJECTIVES datasets, but occurredlater for F OOD (likely due to the smaller number of gradient updates per epoch).

Original: In A.3 we show plotsthat demonstrate that the convergence criterion as defined in §3.3.2 worked reliably for all datasets.
Revised: In B.1 we show plotsthat demonstrate that the convergence criterion as defined in §3.3.2 worked reliably for all datasets.

Original: [Figure]
Revised: [Figure]

Original: One of the benefits of SPoSE is the interpretability of the dimensions of its concept embeddings whichwas evidenced through experiments with humans [18].
Revised: One of the benefits of SPoSE is the interpretability of the dimensions of its concept embeddings whichwas evidenced through experiments with humans [18].

Original: VICE dimensions are equally interpretabledue to similar constraints on its embedding space.
Revised: VICE dimensions are equally interpretabledue to similar constraints on its embedding space.

Original: Therefore, it is easy to sort objects within adimension of the VICE mean embedding matrix µ in descending order, to obtain human judgmentsof what an embedding dimension represents.
Revised: Therefore, it is easy to sort objects within adimension of the VICE mean embedding matrix µ in descending order, to obtain human judgmentsof what an embedding dimension represents.

Original: To illustrate the interpretability of VICE we show in
Revised: To illustrate the interpretability of VICE we show in

Revised: Figure 3 four example dimensions of a representative model for T HINGS , F OOD , and A DJECTIVES .

Original: For T HINGS the dimensions appear to represent categorical , functional , structural , and color-relatedinformation.
Revised: For T HINGS the dimensions appear to represent categorical , functional , structural , and color-relatedinformation.

Original: For F OOD the dimensions appear to be a combination of fried , vegetable , fruit , and sweetfood items.
Revised: For F OOD the dimensions appear to be a combination of fried , vegetable , fruit , and sweetfood items.

Original: For A DJECTIVES the dimensions seem to reflect size/magnitude , visual appearance ,negative valence , and sensory adjectives.
Revised: For A DJECTIVES the dimensions seem to reflect size/magnitude , visual appearance ,negative valence , and sensory adjectives.

Original: In A.9 we show every VICE dimension for T HINGS only,since manuscripts about the other datasets are still in preparation.
Revised: In G we show every VICE dimension for T HINGS only,since manuscripts about the other datasets are still in preparation.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper we introduced VICE, a novel VI approach for learning interpretable object conceptembeddings by modeling human behavior in an odd-one-out triplet task.
Revised: In this paper we introduced VICE, a novel VI approach for learning interpretable object conceptembeddings by modeling human behavior in an odd-one-out triplet task.

Original: We showed that VICEpredicts human behavior close to the estimated best attainable performance across three datasets andthat
Revised: We showed that VICEpredicts human behavior close to the estimated best attainable performance across three datasets andthat

Original: VICE outperformed a competing method, SPoSE, in low sample regimes.
Revised: VICE outperformed a competing method, SPoSE, in low sample regimes.

Original: In addition, VICEhas several characteristics that are desirable for scientific use.
Revised: In addition, VICEhas several characteristics that are desirable for scientific use.

Original: It has an automated procedure fordetermining the number of dimensions needed to explain the data, which further enables the detectionof convergence during training.
Revised: It has an automated procedure fordetermining the number of dimensions needed to explain the data, which further enables the detectionof convergence during training.

Original: This leads to better model reproducibility across different randominitializations and hyperparameter settings.
Revised: This leads to better model reproducibility across different randominitializations and hyperparameter settings.

Original: As a result, VICE can be used out-of-the-box withoutthe necessity to perform an extensive search over random seeds or tuning model hyperparameters.
Revised: As a result, VICE can be used out-of-the-box withoutthe necessity to perform an extensive search over random seeds or tuning model hyperparameters.

Original: Finally, we introduced a PAC learning bound on the generalization performance for a VICE model.
Revised: Finally, we introduced a PAC learning bound on the generalization performance for a VICE model.

Original: One of the limitations of VICE is that it assumes a shared mental representation across participants,akin to much of the literature.
Revised: One of the limitations of VICE is that it assumes a shared mental representation across participants,akin to much of the literature.

Original: However, the VI framework can be leveraged to model inter-individualdifferences, which we plan to do in future work.
Revised: However, the VI framework can be leveraged to model inter-individualdifferences, which we plan to do in future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See § 5.
Revised: [Yes] See § 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main

Original: experimental results (either in the supplemental material or as a URL)?
Revised: experimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: See https://anonymous.4open.science/r/VICE-59F0 for our codebase.
Revised: See https://anonymous.4open.science/r/VICE-59F0 for our codebase.

Original: Seehttps://osf.io/jum2f/ and https://things-initiative.org/ for data.
Revised: Seehttps://osf.io/jum2f/ and https://things-initiative.org/ for data.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See § 4.1, § 4.2 and A.5.
Revised: [Yes] See § 4.1, § 4.2 and D.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] See Figure 2 and Table 1.
Revised: [Yes] See Figure 2 and Table 1.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See A.
Revised: [Yes] See D

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: See https://anonymous.4open.science/r/VICE-59F0 .
Revised: See https://anonymous.4open.science/r/VICE-59F0 .

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See A.8.
Revised: [Yes] See F.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: We re-used crowdsourced data collected by various collaborators, and attribute it to them inthe main paper.
Revised: We re-used crowdsourced data collected by various collaborators, and attribute it to them inthe main paper.

Original: Even though we did not collect these data ourselves, we provide a summaryof the information asked for in 5.
Revised: Even though we did not collect these data ourselves, we provide a summaryof the information asked for in 5.

Original: in A.8, as well as a short description of the task and stimuliin Section 3.1.
Revised: in F, as well as a short description of the task and stimuliin Section 3.1.