Original: VICE: Variational Interpretable Concept Embeddings
Revised: VICE: Variational Interpretable Concept Embeddings

Original: Abstract
Revised: Abstract

Original: A central goal in the cognitive sciences is the development of numerical models for mental representations of object concepts.
Revised: A central goal in the cognitive sciences is the development of numerical models for mental representations of object concepts.

Original: This paper introduces Variational
Revised: This paper introduces Variational

Original: Interpretable Concept Embeddings (VICE), an approximate Bayesian method for embedding object concepts in a vector space using data collected from humans in a triplet odd-one-out task.
Revised: Interpretable Concept Embeddings (VICE), an approximate Bayesian method for embedding object concepts in a vector space using data collected from humans in a triplet odd-one-out task.

Original: VICE uses variational inference to obtain sparse, non-negative representations of object concepts with uncertainty estimates for the embedding values.
Revised: VICE uses variational inference to obtain sparse, non-negative representations of object concepts with uncertainty estimates for the embedding values.

Original: These estimates are used to automatically select the dimensions that best explain the data.
Revised: These estimates are used to automatically select the dimensions that best explain the data.

Original: We derive a PAC learning bound for VICE that can be used to estimate generalization performance or determine a sufficient sample size for experimental design.
Revised: We derive a PAC learning bound for VICE that can be used to estimate generalization performance or determine a sufficient sample size for experimental design.

Original: VICE rivals or outperforms its predecessor, SPoSE, at predicting human behavior in the triplet odd-one-out task.
Revised: VICE rivals or outperforms its predecessor, SPoSE, at predicting human behavior in the triplet odd-one-out task.

Original: Furthermore, VICE’s object representations are more reproducible and consistent across random initializations, highlighting the unique advantage of using VICE for deriving interpretable embeddings from human behavior.
Revised: Furthermore, VICE’s object representations are more reproducible and consistent across random initializations, highlighting the unique advantage of using VICE for deriving interpretable embeddings from human behavior.

Original:  Introduction
Revised:  Introduction

Original: Human knowledge about object concepts encompasses many types of information, including function or purpose, visual appearance, encyclopedic facts, or taxonomic characteristics.
Revised: Human knowledge about object concepts encompasses many types of information, including function or purpose, visual appearance, encyclopedic facts, or taxonomic characteristics.

Original: A central question in cognitive science concerns the representation of this knowledge and its use across different tasks.
Revised: A central question in cognitive science concerns the representation of this knowledge and its use across different tasks.

Original: One approach to this question is inductive and lets subjects list properties for hundreds to thousands of objects [30, 10, 6, 22] yielding large lists of responses about different types of properties.
Revised: One approach to this question is inductive and lets subjects list properties for hundreds to thousands of objects [28, 10, 6, 20] yielding large lists of responses about different types of properties.

Original: Specifically, objects are represented as vectors of binary properties.
Revised: Specifically, objects are represented as vectors of binary properties.

Original: While this approach is agnostic to downstream prediction tasks, it may be biased since subjects may leave out important features and/or mention unimportant ones.
Revised: While this approach is agnostic to downstream prediction tasks, it may be biased since subjects may leave out important features and/or mention unimportant ones.

Original: For example, one may forego a general property (e.g., “is an animal”) while providing a highly specific fact (e.g., “is found in Florida”).
Revised: For example, one may forego a general property (e.g., “is an animal”) while providing a highly specific fact (e.g., “is found in Florida”).

Original: In an alternative, deductive approach, researchers postulate dimensions of interest and subsequently let subjects rate objects in each dimension.
Revised: In an alternative, deductive approach, researchers postulate dimensions of interest and subsequently let subjects rate objects in each dimension.

Original: Binder et al.
Revised: Binder et al.

Original: [3] employed such an approach and collected ratings for hundreds of objects, verbs, and adjectives.
Revised: [3] employed such an approach and collected ratings for hundreds of objects, verbs, and adjectives.

Original: These ratings were gathered over 65 dimensions, reflecting sensory, motor, spatial, temporal, affective, social, and cognitive experiences.
Revised: These ratings were gathered over 65 dimensions, reflecting sensory, motor, spatial, temporal, affective, social, and cognitive experiences.

Original: Nonetheless, it is desirable to discover object representations that are not biased by the conducted behavioral task and whose dimensions are interpretable without necessitating a priori assumptions about their semantic content.
Revised: Nonetheless, it is desirable to discover object representations that are not biased by the conducted behavioral task and whose dimensions are interpretable without necessitating a priori assumptions about their semantic content.

Original: Recently, Zheng et al.
Revised: Recently, Zheng et al.

Original: [50] and Hebart et al.
Revised: [46] and Hebart et al.

Original: [21] introduced SPoSE, a model of the mental representations of 1,854 objects in a 49-dimensional space.
Revised: [19] introduced SPoSE, a model of the mental representations of 1,854 objects in a 49-dimensional space.

Original: The model was learned from human judgments about object similarity, where subjects were asked to determine an odd-one-out object in random triplets of objects.
Revised: The model was learned from human judgments about object similarity, where subjects were asked to determine an odd-one-out object in random triplets of objects.

Original: SPoSE embedded each object in a vector space so that each dimension is non-negative and sparse (most objects have a close-to-zero entry for a given dimension).
Revised: SPoSE embedded each object in a vector space so that each dimension is non-negative and sparse (most objects have a close-to-zero entry for a given dimension).

Original: The authors showed that the embedding dimensions of objects were interpretable and that subjects could coherently label what the dimensions were “about,” ranging from categorical (e.g., animate, food) to functional (e.g., tool), structural (e.g., made of metal or wood), or visual (e.g., coarse pattern).
Revised: The authors showed that the embedding dimensions of objects were interpretable and that subjects could coherently label what the dimensions were “about,” ranging from categorical (e.g., animate, food) to functional (e.g., tool), structural (e.g., made of metal or wood), or visual (e.g., coarse pattern).

Original: The authors hypothesized that interpretability arose from combining positivity and sparsity constraints so that no object was represented by every dimension, and most dimensions were present for only a few objects.
Revised: The authors hypothesized that interpretability arose from combining positivity and sparsity constraints so that no object was represented by every dimension, and most dimensions were present for only a few objects.

Original: In addition,
Revised: In addition,

Original: SPoSE could predict human judgements close to the estimated best attainable performance [50, 21].
Revised: SPoSE could predict human judgements close to the estimated best attainable performance [46, 19].

Original: Despite its notable performance, SPoSE has several limitations.
Revised: Despite its notable performance, SPoSE has several limitations.

Original: The first stems from the use of an ℓ 1 sparsity penalty to promote interpretability.
Revised: The first stems from the use of an ℓ 1 sparsity penalty to promote interpretability.

Original: In SPoSE, 6 to 11 dominant dimensions for an object account for most of the prediction performance.
Revised: In SPoSE, 6 to 11 dominant dimensions for an object account for most of the prediction performance.

Original: These dimensions are different between objects.
Revised: These dimensions are different between objects.

Original: A potential issue with enforcing SPoSE to have even fewer dimensions is that it may cause excessive shrinkage of the dominant values [1].
Revised: A potential issue with enforcing SPoSE to have even fewer dimensions is that it may cause excessive shrinkage of the dominant values [1].

Original: Second, when inspecting the distributions of values across objects, most SPoSE dimensions do not reflect the exponential prior induced by the ℓ 1 penalty.
Revised: Second, when inspecting the distributions of values across objects, most SPoSE dimensions do not reflect the exponential prior induced by the ℓ 1 penalty.

Original: Overcoming this prior may lead to suboptimal performance and inconsistent solutions, specifically in low data regimes.
Revised: Overcoming this prior may lead to suboptimal performance and inconsistent solutions, specifically in low data regimes.

Original: Third, SPoSE uses an ad-hoc criterion for determining the dimensionality of the solution via an arbitrary threshold on the density of each dimension.
Revised: Third, SPoSE uses an ad-hoc criterion for determining the dimensionality of the solution via an arbitrary threshold on the density of each dimension.

Original: Finally, SPoSE has no criterion for determining convergence of its optimization process, nor does it provide any formal guarantees on the sample size needed to learn a model of desired complexity.
Revised: Finally, SPoSE has no criterion for determining convergence of its optimization process, nor does it provide any formal guarantees on the sample size needed to learn a model of desired complexity.

Original: To overcome these limitations we introduce VICE, a variational inference (VI) method for embedding object concepts with interpretable, sparse, and non-negative dimensions.
Revised: To overcome these limitations we introduce VICE, a variational inference (VI) method for embedding object concepts with interpretable, sparse, and non-negative dimensions.

Original: We start by discussing related work and a description of the triplet task and SPoSE, followed by our presentation of theory and experimental results.
Revised: We start by discussing related work and a description of the triplet task and SPoSE, followed by our presentation of theory and experimental results.

Original: Contribution 1: VICE solves major limitations of SPoSE First, VICE encourages shrinkage while allowing for small entry values by using a spike-and-slab prior [5, 12, 15, 31, 38, 43, 48].
Revised: Contribution 1: VICE solves major limitations of SPoSE First, VICE encourages shrinkage while allowing for small entry values by using a spike-and-slab prior [5, 12, 15, 29, 36, 40, 44].

Original: We deem this more appropriate than an exponential prior, because importance – the value an object takes in a dimension – is different from relevance – whether the dimension is applicable to that object – and both can be controlled separately.
Revised: We deem this more appropriate than an exponential prior, because importance – the value an object takes in a dimension – is different from relevance – whether the dimension is applicable to that object – and both can be controlled separately.

Original: Second, we use VI with a unimodal posterior for representing each object in a dimension which yields a mean value and an uncertainty estimate.
Revised: Second, we use VI with a unimodal posterior for representing each object in a dimension which yields a mean value and an uncertainty estimate.

Original: While unimodality makes it possible to use the mean values as representative object embeddings, the uncertainty estimates allow us to use a statistical procedure to automatically select the dimensions that best explain the data.
Revised: While unimodality makes it possible to use the mean values as representative object embeddings, the uncertainty estimates allow us to use a statistical procedure to automatically select the dimensions that best explain the data.

Original: Third, we use this procedure to introduce a convergence criterion that reliably identifies representational stability , i.e. the consistency in the number of selected dimensions.
Revised: Third, we use this procedure to introduce a convergence criterion that reliably identifies representational stability , i.e. the consistency in the number of selected dimensions.

Original: Contribution 2: A PAC bound on the generalization of SPoSE and VICE models This bound can be used retrospectively to provide guarantees about the generalization performance of a converged model.
Revised: Contribution 2: A PAC bound on the generalization of SPoSE and VICE models This bound can be used retrospectively to provide guarantees about the generalization performance of a converged model.

Original: Furthermore, it can be used prospectively to determine the sample size required to identify a representation given the number of objects and a maximum possible number of dimensions.
Revised: Furthermore, it can be used prospectively to determine the sample size required to identify a representation given the number of objects and a maximum possible number of dimensions.

Original: Contribution 3: Extensive evaluation of model performance across multiple datasets We compare
Revised: Contribution 3: Extensive evaluation of model performance across multiple datasets We compare

Original: VICE with SPoSE over three different datasets.
Revised: VICE with SPoSE over three different datasets.

Original: One of these datasets contains concrete objects, another consists of adjectives, and the third is composed of food items.
Revised: One of these datasets contains concrete objects, another consists of adjectives, and the third is composed of food items.

Original: Experimentally, we find that
Revised: Experimentally, we find that

Original: VICE rivals or outperforms the performance of SPoSE in modeling human behavior.
Revised: VICE rivals or outperforms the performance of SPoSE in modeling human behavior.

Original: Moreover, we find that VICE yields dimensions that are much more reproducible, with a lower variance for the number of (selected) dimensions.
Revised: Moreover, we find that VICE yields dimensions that are much more reproducible, with a lower variance for the number of (selected) dimensions.

Original: Both of these measures are particularly important in the cognitive sciences.
Revised: Both of these measures are particularly important in the cognitive sciences.

Original: Lastly, we compare VICE with SPoSE for reduced amounts of data and show that VICE has significantly better performance on all measures.
Revised: Lastly, we compare VICE with SPoSE for reduced amounts of data and show that VICE has significantly better performance on all measures.

Original: A PyTorch implementation of VICE featuring Continuous Integration is publicly available at https: //github.com/
Revised: A PyTorch implementation of VICE featuring Continuous Integration is publicly available at https: //github.com/

Original: LukasMut/VICE .
Revised: LukasMut/VICE .

Original: The GitHub repository additionally includes code to reproduce all of the experiments that are presented in this paper.
Revised: The GitHub repository additionally includes code to reproduce all of the experiments that are presented in this paper.

Original:  Related work
Revised:  Related work

Original: Navarro & Griffiths [34] introduced a method for learning semantic concept embeddings from item similarity data.
Revised: Navarro & Griffiths [32] introduced a method for learning semantic concept embeddings from item similarity data.

Original: The method infers the number of embedding dimensions using the Indian
Revised: The method infers the number of embedding dimensions using the Indian

Original: Buffet Process (IBP) [18].
Revised: Buffet Process (IBP) [18].

Original: Their approach relies on continuous-valued similarity ratings rather than discrete forced-choice behavior and is not directly applicable to our setting.
Revised: Their approach relies on continuous-valued similarity ratings rather than discrete forced-choice behavior and is not directly applicable to our setting.

Original: It is also challenging to scale the IBP to the number of dimensions and samples in our work [39].
Revised: It is also challenging to scale the IBP to the number of dimensions and samples in our work [37].

Original: Roads & Love [41] introduced a method for learning object embeddings from behavior in an 8-rank-2 task.
Revised: Roads & Love [39] introduced a method for learning object embeddings from behavior in an 8-rank-2 task.

Original: Their method predicts behavior from the embeddings by using active sampling [16] to query subjects with the most informative stimuli, yielding an object similarity matrix.
Revised: Their method predicts behavior from the embeddings by using active sampling [16] to query subjects with the most informative stimuli, yielding an object similarity matrix.

Original: The interpretability of embedding dimensions was not considered in this work.
Revised: The interpretability of embedding dimensions was not considered in this work.

Original: A different approach has been to develop interpretable concept representations from text corpora.
Revised: A different approach has been to develop interpretable concept representations from text corpora.

Original: Early methods used word embeddings with positivity and sparsity constraints [33].
Revised: Early methods used word embeddings with positivity and sparsity constraints [31].

Original: Later work in this direction used topic model representations of Wikipedia articles about objects [37], transformations of word embeddings into sparse non-negative representations [46, 35], or predictions of properties [10] or dimensions [49].
Revised: Later work in this direction used topic model representations of Wikipedia articles about objects [35], transformations of word embeddings into sparse non-negative representations [42, 33], or predictions of properties [10] or dimensions [45].

Original: Others have considered using text corpora in conjunction with imaging data [13, 8].
Revised: Others have considered using text corpora in conjunction with imaging data [13, 8].

Original: Finally, Derby et al.
Revised: Finally, Derby et al.

Original: [9] introduced a neural network function that maps the sparse feature space of a semantic property norm to the dense space of a word embedding, identifying informative combinations of properties and ranking candidate properties for new words.
Revised: [9] introduced a neural network function that maps the sparse feature space of a semantic property norm to the dense space of a word embedding, identifying informative combinations of properties and ranking candidate properties for new words.

Original:  Triplet task
Revised:  Triplet task

Original: The triplet task , also known as the triplet odd-one-out task , is used for discovering object concept embeddings from similarity judgments over a set of m different objects.
Revised: The triplet task , also known as the triplet odd-one-out task , is used for discovering object concept embeddings from similarity judgments over a set of m different objects.

Original: These judgments are collected from human participants who are given queries that consist of a triplet of objects (e.g., { “suit” , “flamingo” , “car” } ).
Revised: These judgments are collected from human participants who are given queries that consist of a triplet of objects (e.g., { “suit” , “flamingo” , “car” } ).

Original: Participants are asked to consider the three pairs in a triplet {{ “suit”,“flamingo” } , { “suit”,“car” } , { “flamingo”,“car” ) } , and to decide which pair is the most similar, leaving the third as the odd-one-out.
Revised: Participants are asked to consider the three pairs in a triplet {{ “suit”,“flamingo” } , { “suit”,“car” } , { “flamingo”,“car” }} , and to decide which pair is the most similar, leaving the third as the odd-one-out.

Original: We assign each object a numerical index, e.g., 1 ← “aardvark” , . . .
Revised: We assign each object a numerical index, e.g., 1 ← “aardvark” , . . .

Original: , 1854 ← “zucchini”.
Revised: , 1854 ← “zucchini”.

Original: Let { y, z } denote the indices in this pair, e.g., { y, z } = { 268 , 609 } for “suit” and “flamingo.”
Revised: Let { y, z } denote the indices in this pair, e.g., { y, z } = { 268 , 609 } for “suit” and “flamingo.”

Original: A dataset D is a set of n ordered pairs of presented triplets and selected pairs.
Revised: A dataset D is a set of n ordered pairs of presented triplets and selected pairs.

Original: That is, D := (cid:0) { i s , j s , k s } , { y s , z s } (cid:1) n s =1 , where { y s , z s } ⊂ { i s , j s , k s } .
Revised: That is, D := (cid:0) { i s , j s , k s } , { y s , z s } (cid:1) n s =1 , where { y s , z s } ⊂ { i s , j s , k s } .

Original: In Appendix A we discuss in detail why the triplet task appears to be a sensible choice for modeling object similarities in humans.
Revised: In Appendix A we discuss in detail why the triplet task appears to be a sensible choice for modeling object similarities in humans.

Original: [Figure]
Revised: [Figure]

Original:  Formal setting
Revised:  Formal setting

Original: Sparse Positive object Similarity Embedding (SPoSE) [50] is an approach for finding interpretable embedding dimensions from the triplet task.
Revised: Sparse Positive object Similarity Embedding (SPoSE) [46] is an approach for finding interpretable embedding dimensions from the triplet task.

Original: It does so by finding an embedding vector x i
Revised: It does so by finding an embedding vector x i

Original: =
Revised: =

Original: [ x i 1 , . . .
Revised: [ x i 1 , . . .

Original: , x id ] for every object i .
Revised: , x id ] for every object i .

Original: Let X denote
Revised: Let X denote

Original: the m × d matrix ( x 1 , . .
Revised: the m × d matrix ( x 1 , . .

Original: x m ) and S :
Revised: x m ) and S :

Original: = XX T be the similarity matrix, where S ij denote its entry at i, j .
Revised: = XX T be the similarity matrix, where S ij denote its entry at i, j .

Original: The probability of choosing { y s , z s } as the most similar pair of objects, given an object triplet { i s , j s , k s } and the embedding matrix X , is modeled as
Revised: The probability of choosing { y s , z s } as the most similar pair of objects, given an object triplet { i s , j s , k s } and the embedding matrix X , is modeled as

Original: [Equation]
Revised: [Equation]

Original: At this point we remark that each triplet { i s , j s , k s } is chosen uniformly at random from the collection of all possible sets of object triplets T .
Revised: At this point we remark that each triplet { i s , j s , k s } is chosen uniformly at random from the collection of all possible sets of object triplets T .

Original: That is, { i 1 , j 1 , k 1 } , . .
Revised: That is, { i 1 , j 1 , k 1 } , . .

Original: , { i n , j n , k n } i.i.d.
Revised: , { i n , j n , k n } i.i.d.

Original: ∼ U ( T ) .
Revised: ∼ U ( T ) .

Original: This is precisely stated in Appendix B.1.
Revised: This is precisely stated in Appendix B.1.

Original: SPoSE uses maximum a posteriori (MAP) estimation with a
Revised: SPoSE uses maximum a posteriori (MAP) estimation with a

Original: Laplace prior under a non-negativity constraint (equivalent to an exponential prior) to find the most likely embedding X for the training data D .
Revised: Laplace prior under a non-negativity constraint (equivalent to an exponential prior) to find the most likely embedding X for the training data D .

Original: This leads to the training objective
Revised: This leads to the training objective

Original: [Equation]
Revised: [Equation]

Original: where log p ( D| X ) = constant + (cid:80) ns =1 log p ( { y s , z s }|{ i s , j s , k s } , X ) (see Appendix B.1 for a full derivation of the log-likelihood function) and λ is determined using cross-validation.
Revised: where log p ( D| X ) = constant + (cid:80) ns =1 log p ( { y s , z s }|{ i s , j s , k s } , X ) (see Appendix B.1 for a full derivation of the log-likelihood function) and λ is determined using cross-validation.

Original:  VICE
Revised:  VICE

Original: In contrast to SPoSE, we use mean-field VI [4] instead of a MAP estimate for approximating the posterior probability p ( X |D ) .
Revised: In contrast to SPoSE, we use mean-field VI [4] instead of a MAP estimate for approximating the posterior probability p ( X |D ) .

Original: In VICE we impose additional constraints on the embedding matrix X by using a prior that encourages shrinkage while allowing entries in X to be close to zero.
Revised: In VICE we impose additional constraints on the embedding matrix X by using a prior that encourages shrinkage while allowing entries in X to be close to zero.

Original: Algorithm
Revised: Algorithm

Original: Input: D , θ, α ▷ Recall that D :=
Revised: Input: D , θ, α ▷ Recall that D :=

Original: ( { i s , j s , k s } , { y s , z s } ) ns =1 , and α is a learning rate θ := { µ, σ } ▷ µ, σ ∈ R m × d for s ∈ { 1 , . . . , n } do ϵ ∼ N (0 , I )
Revised: ( { i s , j s , k s } , { y s , z s } ) ns =1 , and α is a learning rate θ := { µ, σ } ▷ µ, σ ∈ R m × d for s ∈ { 1 , . . . , n } do ϵ ∼ N (0 , I )

Original: ▷ Draw i.i.d Gaussian noise where ϵ ∈ R m × d and ϵ ij i.i.d.
Revised: ▷ Draw i.i.d Gaussian noise where ϵ ∈ R m × d and ϵ ij i.i.d.

Original: ∼ N (0 , 1)
Revised: ∼ N (0 , 1)

Original: X θ,ϵ := µ + σ ⊙ ϵ ▷ Apply reparameterization trick; ⊙ is the Hadamard product
Revised: X θ,ϵ := µ + σ ⊙ ϵ ▷ Apply reparameterization trick; ⊙ is the Hadamard product

Original: S :=
Revised: S :=

Original: [ X θ,ϵ ] + [ X θ,ϵ ] T + ▷ Compute similarity matrix using the non-negative parts of X θ,ϵ
Revised: [ X θ,ϵ ] + [ X θ,ϵ ] T + ▷ Compute similarity matrix using the non-negative parts of X θ,ϵ

Original: L data ( X θ,ϵ ) ≜ log (cid:104) exp( S ys,zs ) exp( S is,js )+exp( S is,ks )+exp( S js,ks ) (cid:105) ▷ Data log-likelihood term
Revised: L data ( X θ,ϵ ) ≜ log (cid:104) exp( S ys,zs ) exp( S is,js )+exp( S is,ks )+exp( S js,ks ) (cid:105) ▷ Data log-likelihood term

Original: L complexity ( X θ,ϵ ) ≜ 1 n (cid:2) log q θ ( X θ,ϵ ) − log p ( X θ,ϵ ) (cid:3)
Revised: L complexity ( X θ,ϵ ) ≜ 1 n (cid:2) log q θ ( X θ,ϵ ) − log p ( X θ,ϵ ) (cid:3)

Original: ▷ KL divergence/Regularization term
Revised: ▷ KL divergence/Regularization term

Original: L total ( X θ,ϵ ) ≜
Revised: L total ( X θ,ϵ ) ≜

Original: L complexity ( X θ,ϵ ) − L data ( X θ,ϵ ) µ ← µ − α ∇ µ L total ( X θ,ϵ )
Revised: L complexity ( X θ,ϵ ) − L data ( X θ,ϵ ) µ ← µ − α ∇ µ L total ( X θ,ϵ )

Original: ▷ Update the embedding means σ ← σ − α ∇ σ L total ( X θ,ϵ )
Revised: ▷ Update the embedding means σ ← σ − α ∇ σ L total ( X θ,ϵ )

Original: ▷ Update the embedding standard deviations end for
Revised: ▷ Update the embedding standard deviations end for

Original: Output: θ ▷ Return the optimized set of model parameters θ
Revised: Output: θ ▷ Return the optimized set of model parameters θ

Original: 5.1 Variational Inference
Revised: 5.1 Variational Inference

Original: For VICE we consider approximating p ( X |D ) with a variational distribution, q θ ( X ) , where q θ ∈ Q , and θ is optimized to minimize the KL divergence to the true posterior, p ( X |D ) .
Revised: For VICE we consider approximating p ( X |D ) with a variational distribution, q θ ( X ) , where q θ ∈ Q , and θ is optimized to minimize the KL divergence to the true posterior, p ( X |D ) .

Original: In VICE the KL divergence objective function (derived in Appendix B.2) is
Revised: In VICE the KL divergence objective function (derived in Appendix B.2) is

Original: [Equation]
Revised: [Equation]

Original: Variational distribution VI requires a choice of a parametric variational distribution q ∈ Q .
Revised: Variational distribution VI requires a choice of a parametric variational distribution q ∈ Q .

Original: For
Revised: For

Original: VICE we use a Gaussian distribution with a diagonal covariance matrix q θ ( X ) =
Revised: VICE we use a Gaussian distribution with a diagonal covariance matrix q θ ( X ) =

Original: N ( µ, diag ( σ 2 )) where θ = { µ, σ } .
Revised: N ( µ, diag ( σ 2 )) where θ = { µ, σ } .

Original: Therefore, each embedding dimension has a mean and a standard deviation .
Revised: Therefore, each embedding dimension has a mean and a standard deviation .

Original: We deem a Gaussian variational distribution appropriate for a variety of reasons.
Revised: We deem a Gaussian variational distribution appropriate for a variety of reasons.

Original: First, under certain conditions, the posterior is Gaussian in the infinite-data limit [26].
Revised: First, under certain conditions, the posterior is Gaussian in the infinite-data limit [24].

Original: Second, a unimodal variational distribution makes it possible to use µ 1 , . . . , µ m as representative object embeddings.
Revised: Second, a unimodal variational distribution makes it possible to use µ 1 , . . . , µ m as representative object embeddings.

Original: A fixed representation is useful for downstream use cases that rely on a single embedding vector for each object.
Revised: A fixed representation is useful for downstream use cases that rely on a single embedding vector for each object.

Original: Use cases range from using embeddings as targets in a regression task over unsupervised clustering of embeddings to interpreting embeddings.
Revised: Use cases range from using embeddings as targets in a regression task over unsupervised clustering of embeddings to interpreting embeddings.

Original: More complex variational families are clearly not as practical for this.
Revised: More complex variational families are clearly not as practical for this.

Original: Third, a Gaussian posterior is a computationally convenient choice.
Revised: Third, a Gaussian posterior is a computationally convenient choice.

Original: Similarly to Titsias & Lázaro-Gredilla [47], we use a Monte Carlo approximation of Equation 2 by sampling a limited number of X s from q θ ( X ) during training.
Revised: Similarly to Titsias & Lázaro-Gredilla [43], we use a Monte Carlo approximation of Equation 2 by sampling a limited number of X s from q θ ( X ) during training.

Original: We generate X with the reparameterization trick [25, 40], X θ,ϵ = µ + σ ⊙ ϵ , where ϵ ∈ R m × d is entrywise N (0 , 1) , and ⊙ denotes the
Revised: We generate X with the reparameterization trick [23, 38], X θ,ϵ = µ + σ ⊙ ϵ , where ϵ ∈ R m × d is entrywise N (0 , 1) , and ⊙ denotes the

Original: Hadamard (element-wise) product.
Revised: Hadamard (element-wise) product.

Original: This leads to the objective
Revised: This leads to the objective

Original: [Equation]
Revised: [Equation]

Original: We apply a ReLU function, denoted by [ · ] + , to the sampled X θ,ϵ values to guarantee that X θ,ϵ ∈
Revised: We apply a ReLU function, denoted by [ · ] + , to the sampled X θ,ϵ values to guarantee that X θ,ϵ ∈

Original: R m × d + .
Revised: R m × d + .

Original: As commonly done in the Dropout and Bayesian Neural Network literature [45, 5, 14, 29], we set R = 1 for computational efficiency during the optimization process.
Revised: As commonly done in the Dropout and Bayesian Neural Network literature [41, 5, 14, 27], we set R = 1 for computational efficiency during the optimization process.

Original: The optimization is outlined for individual triplets (i.e., where B = 1 ) for a single training epoch in Algorithm 1.
Revised: The optimization is outlined for individual triplets (i.e., where B = 1 ) for a single training epoch in Algorithm 1.

Original: Posterior probability estimation Computational efficiency at inference time is not as critical as it is during training.
Revised: Posterior probability estimation Computational efficiency at inference time is not as critical as it is during training.

Original: Therefore, we can get a better estimate of the posterior probability distribution over the three possible odd one-one-out choices by letting R ≫ 1 .
Revised: Therefore, we can get a better estimate of the posterior probability distribution over the three possible odd one-one-out choices by letting R ≫ 1 .

Original: Using the optimized variational posterior, q ˆ θ ( X ) , we approximate the probability distribution with a Monte Carlo estimate [17, 5, 29, 4] from R samples X ( r ) =
Revised: Using the optimized variational posterior, q ˆ θ ( X ) , we approximate the probability distribution with a Monte Carlo estimate [17, 5, 27, 4] from R samples X ( r ) =

Original: X ˆ θ,ϵ ( r ) for r = 1 , . . .
Revised: X ˆ θ,ϵ ( r ) for r = 1 , . . .

Original: , R , yielding
Revised: , R , yielding

Original: [Equation]
Revised: [Equation]

Original: Spike-and-slab prior As discussed above, SPoSE induces sparsity through an ℓ 1 penalty which, along with the non-negativity constraint, is equivalent to using an exponential prior.
Revised: Spike-and-slab prior As discussed above, SPoSE induces sparsity through an ℓ 1 penalty which, along with the non-negativity constraint, is equivalent to using an exponential prior.

Original: Through examination of the publicly available histograms of weight values in the two most important SPoSE dimensions (see Figure 5 in Appendix C.3), we observed that the dimensions did not resemble an exponential distribution.
Revised: Through examination of the publicly available histograms of weight values in the two most important SPoSE dimensions (see Figure 2 in Appendix C.3), we observed that the dimensions did not resemble an exponential distribution.

Original: Instead, they contained a spike of probability at zero and a wide slab of probability for the non-zero values.
Revised: Instead, they contained a spike of probability at zero and a wide slab of probability for the non-zero values.

Original: To model this, we use a spike-and-slab Gaussian mixture prior [5, 12, 15, 23, 28],
Revised: To model this, we use a spike-and-slab Gaussian mixture prior [5, 12, 15, 21, 26],

Original: [Equation]
Revised: [Equation]

Original: which encourages shrinkage
Revised: which encourages shrinkage

Original: This prior has three parameters, σ spike , σ slab , and π spike .
Revised: This prior has three parameters, σ spike , σ slab , and π spike .

Original: π spike is the probability that an embedding dimension is drawn from the spike Gaussian.
Revised: π spike is the probability that an embedding dimension is drawn from the spike Gaussian.

Original: Since spike and slab distributions are mathematically interchangeable, by convention we require that σ spike ≪ σ slab .
Revised: Since spike and slab distributions are mathematically interchangeable, by convention we require that σ spike ≪ σ slab .

Original: 5.2 Dimensionality reduction and convergence
Revised: 5.2 Dimensionality reduction and convergence

Original: For interpretability purposes it is desirable for the object embedding dimensionality, d , to be small.
Revised: For interpretability purposes it is desirable for the object embedding dimensionality, d , to be small.

Original: In contrast to SPoSE, which employs a user-defined threshold to prune dimensions, VICE exploits the uncertainty estimates for embedding values to select a subset of informative dimensions.
Revised: In contrast to SPoSE, which employs a user-defined threshold to prune dimensions, VICE exploits the uncertainty estimates for embedding values to select a subset of informative dimensions.

Original: The pruning procedure works by assigning importance scores to each of the d dimensions, which reflect the number of objects that we can confidently say have a non-zero weight in a dimension.
Revised: The pruning procedure works by assigning importance scores to each of the d dimensions, which reflect the number of objects that we can confidently say have a non-zero weight in a dimension.

Original: To compute the score, we use the variational embedding for each object i and dimension j – location µ ij and scale σ ij parameters – to compute the posterior probability that the weight is truncated to zero according to the left tail of a Gaussian distribution with that location and scale (see §5.1).
Revised: To compute the score, we use the variational embedding for each object i and dimension j – location µ ij and scale σ ij parameters – to compute the posterior probability that the weight is truncated to zero according to the left tail of a Gaussian distribution with that location and scale (see §5.1).

Original: This gives us a posterior probability of the weight being zero for each object within a dimension [17].
Revised: This gives us a posterior probability of the weight being zero for each object within a dimension [17].

Original: To calculate the overall importance of a dimension, we estimate the number of objects that have non-zero weights, while controlling the False Discovery Rate [2] with α = 0 .
Revised: To calculate the overall importance of a dimension, we estimate the number of objects that have non-zero weights, while controlling the False Discovery Rate [2] with α = 0 .

Original: We define the importance of each dimension j to be the number of objects for which P ( X ij > 0) ≥ .
Revised: We define the importance of each dimension j to be the number of objects for which P ( X ij > 0) ≥ .

Original: 95 holds.
Revised: 95 holds.

Original: After convergence, we prune the model by removing dimensions with 5 or fewer statistically significant objects.
Revised: After convergence, we prune the model by removing dimensions with 5 or fewer statistically significant objects.

Original: This is a commonly used reliability threshold in semantic property norms (e.g., [30, 10]).
Revised: This is a commonly used reliability threshold in semantic property norms (e.g., [28, 10]).

Original: In gradient-based optimization, the gradient of an objective function with respect to the parameters of a model, ∇L ( θ ) , is used to iteratively find parameters ˆ θ that minimize that function.
Revised: In gradient-based optimization, the gradient of an objective function with respect to the parameters of a model, ∇L ( θ ) , is used to iteratively find parameters ˆ θ that minimize that function.

Original: We use a representational stability criterion to determine convergence.
Revised: We use a representational stability criterion to determine convergence.

Original: That is, the optimization process halts when the number of identified dimensions - as described above - has not changed by a single dimension over the past L epochs (e.g., L = 500 ).
Revised: That is, the optimization process halts when the number of identified dimensions - as described above - has not changed by a single dimension over the past L epochs (e.g., L = 500 ).

Original: Given that our goal is to find stable estimates of the number of dimensions, we considered this to be more appropriate than other convergence criteria such as evaluating the cross-entropy error on a validation set or evidence-based criteria [27, 11].
Revised: Given that our goal is to find stable estimates of the number of dimensions, we considered this to be more appropriate than other convergence criteria such as evaluating the cross-entropy error on a validation set or evidence-based criteria [25, 11].

Original: For further details on convergence and the optimization process see Appendix C.1.
Revised: For further details on convergence and the optimization process see Appendix C.1.

Original:  Sample complexity bound
Revised:  Sample complexity bound

Original: We use statistical learning theory to obtain estimates of the sample size needed to appropriately constrain VICE (and SPoSE) models.
Revised: We use statistical learning theory to obtain estimates of the sample size needed to appropriately constrain VICE (and SPoSE) models.

Original: These estimates can be used retrospectively to obtain probabilistic guarantees on the generalization of a trained model to unseen data, or prospectively to decide how much data to collect for a study.
Revised: These estimates can be used retrospectively to obtain probabilistic guarantees on the generalization of a trained model to unseen data, or prospectively to decide how much data to collect for a study.

Original: The bound presented here applies to any embedding matrix, X , using the model in Equation 1.
Revised: The bound presented here applies to any embedding matrix, X , using the model in Equation 1.

Original: When using this bound to analyze VICE, we extract µ and use it as a fixed embedding to predict the most likely odd-one-out for a query triplet, rather than sampling from the variational distribution.
Revised: When using this bound to analyze VICE, we extract µ and use it as a fixed embedding to predict the most likely odd-one-out for a query triplet, rather than sampling from the variational distribution.

Original: To obtain a useful bound we make two assumptions.
Revised: To obtain a useful bound we make two assumptions.

Original: First, we assume that there exists an upper bound M on the largest value in the embedding.
Revised: First, we assume that there exists an upper bound M on the largest value in the embedding.

Original: Second, we assume that the embeddings obtained by either SPoSE or VICE can be quantized coarsely with marginal losses in predictive performance.
Revised: Second, we assume that the embeddings obtained by either SPoSE or VICE can be quantized coarsely with marginal losses in predictive performance.

Original: We can choose a discretization scale ∆ , e.g., ∆ = 0 . 5 , and round embedding values to a non-negative integer multiple of ∆ .
Revised: We can choose a discretization scale ∆ , e.g., ∆ = 0 . 5 , and round embedding values to a non-negative integer multiple of ∆ .

Original: While we employ this quantization primarily to use learning theory bounds for finite hypothesis classes, it could have benefits for interpretation as well (e.g., a dimension could consist of labels such as zero (0), very low (0.5), low (1), medium (1.5), high (2)).
Revised: While we employ this quantization primarily to use learning theory bounds for finite hypothesis classes, it could have benefits for interpretation as well (e.g., a dimension could consist of labels such as zero (0), very low (0.5), low (1), medium (1.5), high (2)).

Original: In all datasets we have used, SPoSE and VICE embeddings with reasonable priors had values below 2 .
Revised: In all datasets we have used, SPoSE and VICE embeddings with reasonable priors had values below 2 .

Original: 7 (see
Revised: 7 (see

Original: Appendix F.2).
Revised: Appendix F.2).

Original: Given an upper bound M and a discretization scale ∆ , all entries of X are limited to the set A := { 0 , ∆ , . . .
Revised: Given an upper bound M and a discretization scale ∆ , all entries of X are limited to the set A := { 0 , ∆ , . . .

Original: , ( k − 1)∆ , k ∆ } .
Revised: , ( k − 1)∆ , k ∆ } .

Original: The following bound tells us that, given enough samples, the true error rate is not much worse than the estimated error rate for such discretized X matrices, with high probability.
Revised: The following bound tells us that, given enough samples, the true error rate is not much worse than the estimated error rate for such discretized X matrices, with high probability.

Original: Proposition 6.1.
Revised: Proposition 6.1.

Original: Given δ > 0 , ϵ > 0 , and n ≥ ( md log( k + 1) + log (1 /δ )) / (cid:0) 2 ϵ 2 (cid:1) training samples ( { y s , z s } , { i s , j s , k s } ) ns =1 , then
Revised: Given δ > 0 , ϵ > 0 , and n ≥ ( md log( k + 1) + log (1 /δ )) / (cid:0) 2 ϵ 2 (cid:1) training samples ( { y s , z s } , { i s , j s , k s } ) ns =1 , then

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: R ( X
Revised: R ( X

Original: [Equation]
Revised: [Equation]

Original: := P
Revised: := P

Original: [Equation] [Equation][Equation]
Revised: [Equation] [Equation]

Original: z

Original: [Equation]
Revised: [Equation]

Original: { i
Revised: { i

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: This bound follows from applying Hoeffding’s inequality to the ( k + 1) m × d elements of A m × d combined with a union bound.
Revised: This bound follows from applying Hoeffding’s inequality to the ( k + 1) m × d elements of A m × d combined with a union bound.

Original: This proof is virtually identical to that of Theorem 2.13 in
Revised: This proof is virtually identical to that of Theorem 2.13 in

Original: Mohri et al.
Revised: Mohri et al.

Original: A complete proof can be found in Appendix D.
Revised: A complete proof can be found in Appendix D.

Original: To use the bound we first decide on the number of quantization steps k = ⌈ M/ ∆ ⌉ .
Revised: To use the bound we first decide on the number of quantization steps k = ⌈ M/ ∆ ⌉ .

Original: The probability of violating the bound, δ , is analogous to the Type I error control in hypothesis testing and is often set to 0 .
Revised: The probability of violating the bound, δ , is analogous to the Type I error control in hypothesis testing and is often set to 0 .

Original: Since n in our proposition depends very weakly on δ , it is convenient to use δ = 0 .
Revised: Since n in our proposition depends very weakly on δ , it is convenient to use δ = 0 .

Original: The error tolerance ϵ does have a major effect on the sample size estimate provided by the bound.
Revised: The error tolerance ϵ does have a major effect on the sample size estimate provided by the bound.

Original: For the bound to be practically useful, ϵ has to be smaller than the difference between the training error – average zero-one loss over the training set examples – and random guessing error.
Revised: For the bound to be practically useful, ϵ has to be smaller than the difference between the training error – average zero-one loss over the training set examples – and random guessing error.

Original: For the datasets we use in this paper we observed a difference of 0 .
Revised: For the datasets we use in this paper we observed a difference of 0 .

Original: 2 − 0 .
Revised: 2 − 0 .

Original: A conservative choice would halve the accuracy gap of 0.2, giving ϵ = 0 . 1 .
Revised: A conservative choice would halve the accuracy gap of 0.2, giving ϵ = 0 . 1 .

Original: Together, with k = 4 (from above), these values result in a prospective sample size of n ≥ 50 · ( md log(5) + log(1000)) .
Revised: Together, with k = 4 (from above), these values result in a prospective sample size of n ≥ 50 · ( md log(5) + log(1000)) .

Original: To use the bound retrospectively, we fix n , δ while varying ∆ (and consequently k ) to get a guarantee on ϵ , which in turn yields a probabilistic upper bound on the generalization error for an embedding, R ( X ) ≤ ˆ R ( X ) + ϵ .
Revised: To use the bound retrospectively, we fix n , δ while varying ∆ (and consequently k ) to get a guarantee on ϵ , which in turn yields a probabilistic upper bound on the generalization error for an embedding, R ( X ) ≤ ˆ R ( X ) + ϵ .

Original: For more details, see Algorithm 2 in Appendix F.1.
Revised: For more details, see Algorithm 1 in Appendix F.1.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: K L D i v e r g e n c e
Revised: K L D i v e r g e n c e

Original: VICESPoSErandomlower-
Revised: VICESPoSErandomlower-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A cc u r a c y
Revised: A cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D i m e n s i o n a li t y
Revised: D i m e n s i o n a li t y

Original: VICESPoSE
Revised: VICESPoSE

Original:  Experiments
Revised:  Experiments

Original: Data We performed experiments for three triplet datasets: T HINGS (used to develop SPoSE [50, 21]),
Revised: Data We performed experiments for three triplet datasets: T HINGS (used to develop SPoSE [46, 19]),

Original: A DJECTIVES 4 , and F OOD [7] 5 .
Revised: A DJECTIVES 4 , and F OOD [7] 5 .

Original: T HINGS and A DJECTIVES contain random samples from all possible triplet combinations for 1,854 objects and 2,372 adjectives, respectively.
Revised: T HINGS and A DJECTIVES contain random samples from all possible triplet combinations for 1,854 objects and 2,372 adjectives, respectively.

Original: T HINGS and A DJECTIVES are each comprised of a large training dataset which contains no “repeats,” i.e., there is only one human response for each triplet ( ∼ 1 .
Revised: T HINGS and A DJECTIVES are each comprised of a large training dataset which contains no “repeats,” i.e., there is only one human response for each triplet ( ∼ 1 .

Original: 5 M triplets for T HINGS and ∼ 800 K triplets for A DJECTIVES ).
Revised: 5 M triplets for T HINGS and ∼ 800 K triplets for A DJECTIVES ).

Original: For each of these training datasets, 10% of the triplets are assigned to a predefined validation set.
Revised: For each of these training datasets, 10% of the triplets are assigned to a predefined validation set.

Original: The test sets for both contain the results for 1,000 random triplets that are not contained in the training data, with 25 repeats for each triplet.
Revised: The test sets for both contain the results for 1,000 random triplets that are not contained in the training data, with 25 repeats for each triplet.

Original: F OOD contains data for every possible triplet combination for 36 objects and repeats for some triplets.
Revised: F OOD contains data for every possible triplet combination for 36 objects and repeats for some triplets.

Original: We partitioned this dataset into train (45%), validation (5%), and test (50%) sets with disjoint triplets.
Revised: We partitioned this dataset into train (45%), validation (5%), and test (50%) sets with disjoint triplets.

Original: Appendix G provides details about stimuli, data collection, and quality control.
Revised: Appendix G provides details about stimuli, data collection, and quality control.

Original: Experimental setup We implemented both SPoSE and VICE in PyTorch [36] using Adam [24] with η = 0 .
Revised: Experimental setup We implemented both SPoSE and VICE in PyTorch [34] using Adam [22] with η = 0 .

Original: To find the best VICE hyperparameter combination, we performed a grid search over σ spike , σ slab , π spike , optimizing Equation 3 and evaluating each model on the validation set.
Revised: To find the best VICE hyperparameter combination, we performed a grid search over σ spike , σ slab , π spike , optimizing Equation 3 and evaluating each model on the validation set.

Original: All experiments were performed over 20 different random initializations.
Revised: All experiments were performed over 20 different random initializations.

Original: For VICE we observed that the hyperparameter values with the lowest average cross-entropy error on the validation set were highly similar across the different datasets (see Table 2 in Appendix E).
Revised: For VICE we observed that the hyperparameter values with the lowest average cross-entropy error on the validation set were highly similar across the different datasets (see Table 1 in Appendix E).

Original: For more information about the experimental setup, including details about training, weight initialization, hyperparameter grid, and optimal combination, see Appendix E.
Revised: For more information about the experimental setup, including details about training, weight initialization, hyperparameter grid, and optimal combination, see Appendix E.

Original: Evaluation measures VICE estimates p ( { y, z }|{ i, j, k } ) , the model’s softmax probability distribution over a given triplet (see Equation 4).
Revised: Evaluation measures VICE estimates p ( { y, z }|{ i, j, k } ) , the model’s softmax probability distribution over a given triplet (see Equation 4).

Original: The first evaluation measure we consider is the prediction accuracy with respect to the correct human choice { y, z } , where the predicted pair is arg max { y,z } ˆ p ( { y, z }|{ i, j, k } ) .
Revised: The first evaluation measure we consider is the prediction accuracy with respect to the correct human choice { y, z } , where the predicted pair is arg max { y,z } ˆ p ( { y, z }|{ i, j, k } ) .

Original: We estimate an upper bound on the prediction accuracy by using the repeats in the test set (see Appendix E).
Revised: We estimate an upper bound on the prediction accuracy by using the repeats in the test set (see Appendix E).

Original: A Bayes-optimal classifier would predict the majority outcome for any triplet, whereas random guessing gives an accuracy of 1 / 3 .
Revised: A Bayes-optimal classifier would predict the majority outcome for any triplet, whereas random guessing gives an accuracy of 1 / 3 .

Original: Note that the triplet task is subjective and thus there is not necessarily a definitive correct answer for a given triplet.
Revised: Note that the triplet task is subjective and thus there is not necessarily a definitive correct answer for a given triplet.

Original: If a test set contains multiple responses for a triplet, it provides information about the relative similarities of the three object pairs.
Revised: If a test set contains multiple responses for a triplet, it provides information about the relative similarities of the three object pairs.

Original: This lets us calculate probability distributions over answers for each triplet.
Revised: This lets us calculate probability distributions over answers for each triplet.

Original: Approximating these distributions closely is important in cognitive science applications.
Revised: Approximating these distributions closely is important in cognitive science applications.

Original: Therefore, we additionally evaluate the models by computing the KL divergences between the models’ posterior probability estimates ˆ p ( { y, z }|{ i, j, k } ) (see Equation 4) and the human probability distributions inferred from the test set across all test triplets and report the mean.
Revised: Therefore, we additionally evaluate the models by computing the KL divergences between the models’ posterior probability estimates ˆ p ( { y, z }|{ i, j, k } ) (see Equation 4) and the human probability distributions inferred from the test set across all test triplets and report the mean.

Original: 7.1 Results on THINGS
Revised: 7.1 Results on THINGS

Original: Full dataset For this experiment we compared a representative model from VICE and SPoSE.
Revised: Full dataset For this experiment we compared a representative model from VICE and SPoSE.

Original: The representative models were chosen to be those with the median cross-entropy error on the validation set over the 20 seeds.
Revised: The representative models were chosen to be those with the median cross-entropy error on the validation set over the 20 seeds.

Original: For VICE, we set the number of Monte Carlo samples to R = 50 .
Revised: For VICE, we set the number of Monte Carlo samples to R = 50 .

Original: On the test set, VICE and SPoSE achieved similar prediction accuracies of 0 . 638 and 0 .
Revised: On the test set, VICE and SPoSE achieved similar prediction accuracies of 0 . 638 and 0 .

Original: 637 respectively (estimated upper bound for accuracy was 0 . 673 ).
Revised: 637 respectively (estimated upper bound for accuracy was 0 . 673 ).

Original: Likewise, VICE and SPoSE achieved similar average KL divergences of 0 . 100 and 0 .
Revised: Likewise, VICE and SPoSE achieved similar average KL divergences of 0 . 100 and 0 .

Original: 103 respectively (random guessing KL divergence was 0 . 366 ).
Revised: 103 respectively (random guessing KL divergence was 0 . 366 ).

Original: The differences between the median model accuracies and KL divergences were not statistically significant according to a two-sided paired t -test over individual test triplets.
Revised: The differences between the median model accuracies and KL divergences were not statistically significant according to a two-sided paired t -test over individual test triplets.

Original: Hence, VICE and SPoSE predicted triplets approximately equally well when trained on the full dataset.
Revised: Hence, VICE and SPoSE predicted triplets approximately equally well when trained on the full dataset.

Original: This is not surprising since Bayesian methods based on Monte Carlo sampling become more like deterministic maximum likelihood estimation in the infinite data limit.
Revised: This is not surprising since Bayesian methods based on Monte Carlo sampling become more like deterministic maximum likelihood estimation in the infinite data limit.

Original: If n → ∞ , then the left-hand side in the expectation of Equation 2, which accounts for the contribution of the prior, goes to zero.
Revised: If n → ∞ , then the left-hand side in the expectation of Equation 2, which accounts for the contribution of the prior, goes to zero.

Original: Conversely, the effects of the prior are more prominent when n is small, as we show in the following section.
Revised: Conversely, the effects of the prior are more prominent when n is small, as we show in the following section.

Original: Data efficiency experiments Performance on small datasets is important in cognitive science, since behavioral experiments often have small sample sizes (e.g., tens to hundreds of volunteer in-lab subjects) compared to THINGS, which is particularly large, or they can be costly to collect.
Revised: Data efficiency experiments Performance on small datasets is important in cognitive science, since behavioral experiments often have small sample sizes (e.g., tens to hundreds of volunteer in-lab subjects) compared to THINGS, which is particularly large, or they can be costly to collect.

Original: To test whether VICE can model the data better than SPoSE in small sample regimes, we performed experiments training on smaller subsets with sizes equal to 5%, 10%, 20%, and 50% of the training dataset.
Revised: To test whether VICE can model the data better than SPoSE in small sample regimes, we performed experiments training on smaller subsets with sizes equal to 5%, 10%, 20%, and 50% of the training dataset.

Original: For each size, we divided the training set into equal partitions and performed an experiment for each partition, thereby giving multiple results for each subset size.
Revised: For each size, we divided the training set into equal partitions and performed an experiment for each partition, thereby giving multiple results for each subset size.

Original: Validation and held-out test sets remained unchanged.
Revised: Validation and held-out test sets remained unchanged.

Original: In Figure 2 we show the average KL divergence (left) and prediction accuracy (middle) across data partitions and 20 random seeds (used to initialize model parameters) for various dataset sizes.
Revised: In Figure 2 we show the average KL divergence (left) and prediction accuracy (middle) across data partitions and 20 random seeds (used to initialize model parameters) for various dataset sizes.

Original: The average performance across partitions was used to compute the confidence intervals (CIs).
Revised: The average performance across partitions was used to compute the confidence intervals (CIs).

Original: The differences in prediction accuracies and KL divergences between VICE and
Revised: The differences in prediction accuracies and KL divergences between VICE and

Original: SPoSE were notable for the 5% and 10% data subsets.
Revised: SPoSE were notable for the 5% and 10% data subsets.

Original: In the former, SPoSE predicted only slightly better than random guessing; in the latter, SPoSE showed a large variability between random seeds and data partitions, as can be seen in the 95% CIs.
Revised: In the former, SPoSE predicted only slightly better than random guessing; in the latter, SPoSE showed a large variability between random seeds and data partitions, as can be seen in the 95% CIs.

Original: In comparison, VICE showed small variation in the two performance metrics across random seeds and performed much better than random guessing.
Revised: In comparison, VICE showed small variation in the two performance metrics across random seeds and performed much better than random guessing.

Original: The differences between VICE and SPoSE for the 5% and 10% subset scenarios were statistically significant according to a two-sided paired t -test comparing individual triplet predictions between median models ( p < 0 .
Revised: The differences between VICE and SPoSE for the 5% and 10% subset scenarios were statistically significant according to a two-sided paired t -test comparing individual triplet predictions between median models ( p < 0 .

Original: For the full training dataset, VICE used significantly fewer dimensions ( p < 0 .
Revised: For the full training dataset, VICE used significantly fewer dimensions ( p < 0 .

Original: 001 ; unpaired sign test) than SPoSE to achieve comparable performance.
Revised: 001 ; unpaired sign test) than SPoSE to achieve comparable performance.

Original: 7.2 Other results
Revised: 7.2 Other results

Original: Other datasets On the A DJECTIVES test set, VICE and SPoSE had accuracies of 0 .
Revised: Other datasets On the A DJECTIVES test set, VICE and SPoSE had accuracies of 0 .

Original: 559 and 0 .
Revised: 559 and 0 .

Original: 562 , respectively (estimated upper bound was 0 . 607 ), and KL divergences of 0 . 083 and 0 .
Revised: 562 , respectively (estimated upper bound was 0 . 607 ), and KL divergences of 0 . 083 and 0 .

Original: 088 , respectively (random guessing was 0 . 366 ).
Revised: 088 , respectively (random guessing was 0 . 366 ).

Original: On the F OOD test set, VICE and SPoSE median models had accuracies of 0 .
Revised: On the F OOD test set, VICE and SPoSE median models had accuracies of 0 .

Original: 693 and 0 .
Revised: 693 and 0 .

Original: 698 , respectively.
Revised: 698 , respectively.

Original: The differences in accuracy (and KL divergence for A DJECTIVES ) between VICE and SPoSE for both datasets were not statistically significant according to two-sided paired t -tests on the median model predictions across triplets on the test set.
Revised: The differences in accuracy (and KL divergence for A DJECTIVES ) between VICE and SPoSE for both datasets were not statistically significant according to two-sided paired t -tests on the median model predictions across triplets on the test set.

Original: However, for F OOD ,
Revised: However, for F OOD ,

Original: VICE used significantly fewer dimensions than SPoSE to achieve similar performance (see Table 1).
Revised: VICE used significantly fewer dimensions than SPoSE to achieve similar performance (see Table 1).

Original: Hyperparameters We observed that VICE’s performance is fairly insensitive to hyperparameter selection.
Revised: Hyperparameters We observed that VICE’s performance is fairly insensitive to hyperparameter selection.

Original: Using default hyperparameters, σ spike = 0 .
Revised: Using default hyperparameters, σ spike = 0 .

Original: 25 , σ slab = 1 . 0 , π sigma = 0 .
Revised: 25 , σ slab = 1 . 0 , π sigma = 0 .

Original: 5 , yields an accuracy score within 0 .
Revised: 5 , yields an accuracy score within 0 .

Original: 015 of the best cross-validated model in the full dataset setting, on all three datasets.
Revised: 015 of the best cross-validated model in the full dataset setting, on all three datasets.

Original: 7.3 Reproducibility and Representational stability
Revised: 7.3 Reproducibility and Representational stability

Original: Reproducibility As mentioned in the introduction, a key criterion for learning concept representations beyond predictive performance is reproducibility , i.e., learning similar representations for different random initializations on the same training data.
Revised: Reproducibility As mentioned in the introduction, a key criterion for learning concept representations beyond predictive performance is reproducibility , i.e., learning similar representations for different random initializations on the same training data.

Original: To evaluate this, we compared the learned embeddings from 20 differently initialized VICE and SPoSE models.
Revised: To evaluate this, we compared the learned embeddings from 20 differently initialized VICE and SPoSE models.

Original: The first aspect of reproducibility we investigate is whether the models yield a consistent number of embedding dimensions across random seeds.
Revised: The first aspect of reproducibility we investigate is whether the models yield a consistent number of embedding dimensions across random seeds.

Original: As reported in Table 1, VICE yielded fewer dimensions than SPoSE with less variance in the number of dimensions for all three datasets.
Revised: As reported in Table 1, VICE yielded fewer dimensions than SPoSE with less variance in the number of dimensions for all three datasets.

Original: The embedding dimensionality was significantly smaller according to an independent t -test for T HINGS ( p < 0 .
Revised: The embedding dimensionality was significantly smaller according to an independent t -test for T HINGS ( p < 0 .

Original: 001 ) and F OOD ( p < 0 . 001 ), but not for A DJECTIVES ( p = 0 . 108 ).
Revised: 001 ) and F OOD ( p < 0 . 001 ), but not for A DJECTIVES ( p = 0 . 108 ).

Original: The difference in the standard deviations for A DJECTIVES was statistically significant according to a two-sided F -test ( p = 0 . 002 ), but was not statistically significant for T HINGS ( p = 0 .
Revised: The difference in the standard deviations for A DJECTIVES was statistically significant according to a two-sided F -test ( p = 0 . 002 ), but was not statistically significant for T HINGS ( p = 0 .

Original: 283 ) or F OOD ( p = 0 . 378 ).
Revised: 283 ) or F OOD ( p = 0 . 378 ).

Original: In addition, we observed that the identified dimensionality is consistent as long as d is chosen to be sufficiently large (see Appendix E).
Revised: In addition, we observed that the identified dimensionality is consistent as long as d is chosen to be sufficiently large (see Appendix E).

Original: [Table]
Revised: [Table]

Original: The second aspect to reproducibility we examine is the degree to which the identified dimensions are similar across random initializations up to a permutation.
Revised: The second aspect to reproducibility we examine is the degree to which the identified dimensions are similar across random initializations up to a permutation.

Original: To calculate the number of reproducible dimensions, we associated each embedding dimension of a model with the most similar embedding dimension across the other models.
Revised: To calculate the number of reproducible dimensions, we associated each embedding dimension of a model with the most similar embedding dimension across the other models.

Original: We quantify reproducibility of a dimension as the average Pearson correlation between one dimension and its best match across the 19 remaining models.
Revised: We quantify reproducibility of a dimension as the average Pearson correlation between one dimension and its best match across the 19 remaining models.

Original: In Tablewe report the average relative number of dimensions with a correlation > 0 . 8 across models.
Revised: In Tablewe report the average relative number of dimensions with a correlation > 0 . 8 across models.

Original: The embedding dimensions in VICE were more reproducible than those in SPoSE.
Revised: The embedding dimensions in VICE were more reproducible than those in SPoSE.

Original: The difference in average reproducibility scores was statistically significant according to a two-sided, independent t -test for F OOD ( p = 0 . 030 ) and T HINGS ( p = 0 . 008 ), but not for A DJECTIVES ( p = 0 . 466 ).
Revised: The difference in average reproducibility scores was statistically significant according to a two-sided, independent t -test for F OOD ( p = 0 . 030 ) and T HINGS ( p = 0 . 008 ), but not for A DJECTIVES ( p = 0 . 466 ).

Original: Representational stability For all three datasets, VICE found reproducible and stable dimensions across all 20 random initializations and converged when the embedding dimensionality had not changed over the past L = 500 epochs.
Revised: Representational stability For all three datasets, VICE found reproducible and stable dimensions across all 20 random initializations and converged when the embedding dimensionality had not changed over the past L = 500 epochs.

Original: The median number of epochs to achieve representational stability was comparable for the similarly-sized T HINGS and A DJECTIVES datasets, but occurred later for F OOD (likely due to the smaller number of gradient updates per epoch).
Revised: The median number of epochs to achieve representational stability was comparable for the similarly-sized T HINGS and A DJECTIVES datasets, but occurred later for F OOD (likely due to the smaller number of gradient updates per epoch).

Original: In Appendix C.1 we show plots that demonstrate that the convergence criterion as defined in §5.2 worked reliably for all datasets.
Revised: In Appendix C.1 we show plots that demonstrate that the convergence criterion as defined in §5.2 worked reliably for all datasets.

Original: 7.4 Interpretability
Revised: 7.4 Interpretability

Original: One of the benefits of SPoSE is the interpretability of the dimensions of its concept embeddings which was evidenced through experiments with humans [21].
Revised: One of the benefits of SPoSE is the interpretability of the dimensions of its concept embeddings which was evidenced through experiments with humans [19].

Original: VICE dimensions are equally interpretable due to similar constraints on its embedding space.
Revised: VICE dimensions are equally interpretable due to similar constraints on its embedding space.

Original: Therefore, it is easy to sort objects within a dimension of the VICE mean embedding matrix µ in descending order, to obtain human judgments of what an embedding dimension represents.
Revised: Therefore, it is easy to sort objects within a dimension of the VICE mean embedding matrix µ in descending order, to obtain human judgments of what an embedding dimension represents.

Original: To illustrate the interpretability of VICE we show in
Revised: To illustrate the interpretability of VICE we show in

Original: Figure 3 four example dimensions of a representative model for T HINGS , F OOD , and A DJECTIVES .
Revised: Figure 3 four example dimensions of a representative model for T HINGS , F OOD , and A DJECTIVES .

Original: For T HINGS the dimensions appear to represent categorical , functional , structural , and color-related information.
Revised: For T HINGS the dimensions appear to represent categorical , functional , structural , and color-related information.

Original: For F OOD the dimensions appear to be a combination of fried , vegetable , fruit , and sweet food items.
Revised: For F OOD the dimensions appear to be a combination of fried , vegetable , fruit , and sweet food items.

Original: For A DJECTIVES the dimensions seem to reflect size/magnitude , visual appearance , negative valence , and sensory adjectives.
Revised: For A DJECTIVES the dimensions seem to reflect size/magnitude , visual appearance , negative valence , and sensory adjectives.

Original: In Appendix H we show VICE dimensions for T HINGS only, since manuscripts about the other datasets are still in preparation, and, hence, cannot be shown publicly.
Revised: In Appendix H we show VICE dimensions for T HINGS only, since manuscripts about the other datasets are still in preparation, and, hence, cannot be shown publicly.

Original:  Limitations
Revised:  Limitations

Original: The goal of our method is to identify general mental representations of objects.
Revised: The goal of our method is to identify general mental representations of objects.

Original: The dimensions identified by a model reflect semantic characteristics that explain task performance for many subjects in the experimental subject population (here, Amazon Mechanical Turk subjects in the United States).
Revised: The dimensions identified by a model reflect semantic characteristics that explain task performance for many subjects in the experimental subject population (here, Amazon Mechanical Turk subjects in the United States).

Original: As such, it is possible that they reflect biases widespread in that population.
Revised: As such, it is possible that they reflect biases widespread in that population.

Original: Furthermore, the choice of population may affect the identified dimensions.
Revised: Furthermore, the choice of population may affect the identified dimensions.

Original: That is, a chéf may classify food items differently from a lay subject, and a linguist would likely have a more complex representation of an adjective.
Revised: That is, a chéf may classify food items differently from a lay subject, and a linguist would likely have a more complex representation of an adjective.

Original: The effects of expertise or developmental stage in mental representations are of obvious interest to cognitive scientists.
Revised: The effects of expertise or developmental stage in mental representations are of obvious interest to cognitive scientists.

Original: Therefore, we envision further research in those areas, which may additionally provide some indication of how widely representations can vary.
Revised: Therefore, we envision further research in those areas, which may additionally provide some indication of how widely representations can vary.

Original:  Conclusion
Revised:  Conclusion

Original: One of the central goals in the cognitive sciences is the development of computational models of mental representations of object concepts.
Revised: One of the central goals in the cognitive sciences is the development of computational models of mental representations of object concepts.

Original: Such models may serve as a component for other behavioral prediction models or as a basis for identifying the nature of concept representations in the human brain.
Revised: Such models may serve as a component for other behavioral prediction models or as a basis for identifying the nature of concept representations in the human brain.

Original: In this paper we introduced VICE, a novel VI approach for learning interpretable object concept embeddings by modeling human behavior in an triplet odd-one-out task.
Revised: In this paper we introduced VICE, a novel VI approach for learning interpretable object concept embeddings by modeling human behavior in an triplet odd-one-out task.

Original: We showed that VICE predicts human behavior close to the estimated best attainable performance across three datasets and that VICE outperforms a competing method, SPoSE, in low data regimes.
Revised: We showed that VICE predicts human behavior close to the estimated best attainable performance across three datasets and that VICE outperforms a competing method, SPoSE, in low data regimes.

Original: In addition,
Revised: In addition,

Original: VICE has several characteristics that are desirable for scientific use.
Revised: VICE has several characteristics that are desirable for scientific use.

Original: It has an automated procedure for determining the number of dimensions sufficient to explain the data, which further enables the detection of convergence during training.
Revised: It has an automated procedure for determining the number of dimensions sufficient to explain the data, which further enables the detection of convergence during training.

Original: This leads to better model reproducibility across different random initializations and hyperparameter settings.
Revised: This leads to better model reproducibility across different random initializations and hyperparameter settings.

Original: As a result, VICE can be used out-of-the-box without requiring to perform an extensive search over random seeds or tuning model hyperparameters.
Revised: As a result, VICE can be used out-of-the-box without requiring to perform an extensive search over random seeds or tuning model hyperparameters.

Original: Finally, we introduced a PAC learning bound on the generalization performance for a VICE model.
Revised: Finally, we introduced a PAC learning bound on the generalization performance for a VICE model.

Original: Although VICE assumes a shared mental representation across participants - akin to much of the literature -, we believe that the VI framework can be leveraged to model inter-individual differences, which we plan to do in future work.
Revised: Although VICE assumes a shared mental representation across participants - akin to much of the literature -, we believe that the VI framework can be leveraged to model inter-individual differences, which we plan to do in future work.

Original: Acknowledgments
Revised: Acknowledgments

Original: LM and RV acknowledge support by the Federal Ministry of Education and Research (BMBF) for the Berlin Institute for the Foundations of Learning and Data (BIFOLD) (01IS18037A).
Revised: LM and RV acknowledge support by the Federal Ministry of Education and Research (BMBF) for the Berlin Institute for the Foundations of Learning and Data (BIFOLD) (01IS18037A).

Original: LM and MNH acknowledge support by a Max Planck Research Group grant awarded to MNH by the Max Planck Society.
Revised: LM and MNH acknowledge support by a Max Planck Research Group grant awarded to MNH by the Max Planck Society.

Original: FP, CZ, and PM acknowledge the support of the National Institute of
Revised: FP, CZ, and PM acknowledge the support of the National Institute of

Original: Mental Health Intramural Research Program (ZIC-MH002968).
Revised: Mental Health Intramural Research Program (ZIC-MH002968).

Original: PM acknowledges the support of the
Revised: PM acknowledges the support of the

Original: Naval Postgraduate School’s Research Initiation Program.
Revised: Naval Postgraduate School’s Research Initiation Program.

Original: This study utilized the high-performance computational capabilities of the Biowulf Linux cluster at the National Institutes of Health, Bethesda,
Revised: This study utilized the high-performance computational capabilities of the Biowulf Linux cluster at the National Institutes of Health, Bethesda,

Original: MD (http://biowulf.nih.gov) and the Raven and Cobra Linux clusters at the Max Planck Computing & Data Facility, Garching, Germany (https://www.mpcdf.mpg.de/services/supercomputing/).
Revised: MD (http://biowulf.nih.gov) and the Raven and Cobra Linux clusters at the Max Planck Computing & Data Facility, Garching, Germany (https://www.mpcdf.mpg.de/services/supercomputing/).

Original: The authors would like to thank Chris Baker for useful initial discussions, Shruti Japee, Jason Avery, and
Revised: The authors would like to thank Chris Baker for useful initial discussions, Shruti Japee, Jason Avery, and

Original: Alex Martin for sharing data, and Erik Daxberger, Lorenz Linhardt, Adrian Hill, Niklas Schmitz, and
Revised: Alex Martin for sharing data, and Erik Daxberger, Lorenz Linhardt, Adrian Hill, Niklas Schmitz, and

Original: Marco Morik for valuable feedback on earlier versions of the paper.A Triplet task
Revised: Marco Morik for valuable feedback on earlier versions of the paper.

Original: The triplet task compares favorably to other possible alternatives for eliciting human pairwise object similarity judgments.

Original: Asking for similarity ratings directly, on a numeric or qualitative scale, introduces the difficulty of differing inter-individual calibration of ratings.

Original: Having a choice task reduces this inter-individual variability by reducing the number of possible actions the user can make, e.g., three in the triplet task.

Original: Within choice tasks, there are also k -way forced choice tasks which specify a reference object and query the user for the most similar object to the reference among a list of k candidate objects.

Original: It seems likely that the choice in the forced choice task is not strictly driven by pairwise item similarities between the choices and the reference, but that the reference may influence the selection of features used to assess item similarities.

Original: For instance, the subject may choose the most prominent feature associated with the reference, and then make a uni-dimensional assessment of which candidate scores highest on that feature.

Original: This is less likely to be an issue in the triplet task because with no reference object, the subject has to consider all three pairs of items to evaluate which pair has greater similarity than the other two pairs.

Original: In particular, it would be less feasible to try to find a single feature that would be relevant for all three pairs of objects.

Original: Hence, we speculate that the triplet task may promote more multidimensional similarity assessments than the k -way forced-choice task.

Original: Many studies find it convenient to pool triplet tasks from multiple participants.

Original: This could be justified under an assumption where subjects within a population all share the same common mental representations of items, which in turn determine the pairwise similarities that govern the choice of the odd-one-out selection.

Original: Alternatively, it could be the case that the representation of each item has random variation, both within a subject and across subjects.

Original: In such a scenario, it could be useful to use a model which assumes that a probabilistic (rather than a deterministic ) embedding governs the triplet choices.

Original: As we discussed theoretically in §5 and showed empirically in §7, variational bayesian inference is one way that appears appropriate to model such a probabilistic embedding.

Original: B Objective function B.1 Probability model and the log-likelihood function

Original: In the following we define the log-likelihood function of the data given the embedding matrix, log p ( D| X ) .

Original: Recall that D := (cid:0) { i s , j s , k s } , { y s , z s } (cid:1) n s =1 .

Original: For simplicity, we assume that each triplet { i s , j s , k s } is chosen uniformly at random from the collection of all possible sets of object triplets

Original: T .

Original: That is, { i 1 , j 1 , k 1 } , . .

Original: , { i n , j n , k n } i.i.d.

Original: ∼ U ( T ) .

Original: Hence, the probability for choosing a triplet, p ( { i s , j s , k s } ) , is the same for all D s ∈ {D 1 , . .

Original: , D n } and can therefore be treated as a coefficient for computing p ( D| X ) .

Original: For our probability model the log-likelihood of the data given the embedding matrix, log p ( D| X ) , can then be defined as

Original: [Equation]

Original: Recall that p ( { i s , j s , k s } ) is a coefficient that is the same for all s ∈ { 1 , . . . , n } .

Original: We let C :

Original: =

Original: p ( { i s , j s , k s } ) and thus we can further rewrite, log

Original: [Equation]

Original: where (6) follows from the chain rule of probability.

Original: Note that the constant n log C can be ignored in the minimization of the VICE objective function, whose derivation is outlined in Appendix B.2.

Original: B.2 KL divergence

Original: In VI, one minimizes the KL divergence between q θ ( X ) , the variational posterior, and, p ( X |D ) , the true posterior,

Original: [Equation]

Original: where

Original: [Equation]

Original: X ) − log p ( X ) − log p ( D| X ) (cid:3) + log p ( D ) =

Original: E q θ ( X ) (cid:34) log q θ ( X ) − log p ( X ) − n log C − n (cid:88) s

Original: = log p ( { y s , z s }|{ i s , j s , k s } , X ) (cid:35) + log p ( D )

Original: =

Original: E q θ ( X ) (cid:34) log q θ ( X ) − log p ( X ) − n (cid:88)

Original: s

Original: = log p ( { y s , z s }|{ i s , j s , k s } , X ) (cid:35) − n log C + log p ( D ) .

Original: B.3 VICE objective function

Original: Because n log C and log p ( D ) are constants and not functions of the variational parameters, we can ignore both terms in the minimization and get the following VICE objective

Original: [Equation]

Original: Multiplying this by (1 /n ) , where n is the number of training examples, does not change the minimum of the objective function and results in

Original: [Equation]

Original: C Optimization, convergence and prior C.1 Gradient-based optimization

Original: In gradient-based optimization, the gradient of an objective function with respect to the parameters of a model, ∇L ( θ ) , is used to iteratively find parameters ˆ θ that minimize that function.

Original: Equation computes the expected log-likelihood of the entire training data.

Original: However, using every training data point to compute a gradient update is computationally expensive for large datasets and often generalizes poorly for non-convex objective functions [44].

Original: In VICE, we stochastically approximate the training log-likelihood using random subsets (i.e., mini-batches) of the training data, where each mini-batch consists of B triplets [42].

Original: This leads to an objective function that is a doubly stochastic approximation of Equation 3 [47], due to sampling weights from the variational distribution, q θ ∈ Q , and sampling a random mini-batch of training examples during each gradient step (i.e., performing mini-batch gradient descent),

Original: [Equation]

Original: where p ( { y, z }|{ i, j, k } , X ) for a single sample is defined in Equation 1.

Original: To find parameters, θ , that optimize Equation 3, we iteratively update both the means, µ , and the standard deviations, σ , of each

Original: VICE dimension, by

Original: [Equation]

Original: and

Original: [Equation]

Original: where α is the learning rate for θ .

Original: C.2 Convergence

Original: We define convergence as the point in time, t ∗ , where the number of embedding dimensions has not changed by a single dimension over the past L epochs.

Original: We denote this as the point of representational stability .

Original: To ensure convergence, we recommend letting L be relatively large (e.g., L ≫ 100 ).

Original: We found L = 500 to work well for our experiments.

Original: Figure 4 shows that the convergence criterion defined in §5.2 worked reliably for all datasets.

Original: [Figure]

Original: i

Original: [Figure]

Original: i o n

Original: [Figure]

Original: li t

Original: [Figure]

Original: C.3 (In-)efficient prior choice

Original: As discussed in §5.

Original: [Figure]

Original: SPoSE imposes a combination of an ℓ 1 penalty and a non-negativity constraint on its embedding values.

Original: This is analogous to having an exponential prior on those values.

Original: If we consider the distribution of values across objects for the two most important SPoSE dimensions in

Original: Figure 5, we can see that they show a distribution with a spike around 0 , and a much smaller, wide slab of probability mass for non-zero values.

Original: Overcoming the exponential prior is data inefficient.

Original: SPoSE was developed using a dataset that was orders of magnitude larger than what a typical psychological experiment might collect, a setting in which this issue would likely not manifest itself.

Original: However, SPoSE has not been tested on the more common, smaller datasets

Original: [Equation]

Original: and our results suggest that the implicit SPoSE prior leads to suboptimal results in those regimes, when compared to the spike-and-slab prior used by VICE, as shown in Figure

Original: [Equation]

Original: D Proof of Proposition 3.

Original: Proof.

Original: Recall that

Original: [Equation]

Original: and

Original: [Equation]

Original: Using the union bound we have that

Original: [Equation]

Original: Since E (cid:104) ˆ R ( X ) (cid:105)

Original: =

Original: R ( X ) and the summands of ˆ R ( X ) are independent and satisfy

Original: [Equation]

Original: for all X ∈ A m × d we can apply Hoeffding’s Inequality to yield

Original: [Equation]

Original: for all X ∈ A m × d .

Original: The set A m × d contains ( k + 1) md elements so combining Equation 7 and

Original: Equation 8 we get

Original: [Equation]

Original: To arrive at the proposition statement we need that

Original: [Equation]

Original: Solving for n we get

Original: [Equation]

Original: E Experimental details

Original: Training Although we have developed a reliable convergence criterion for VICE (see §5.2), to guarantee a fair comparison between VICE and SPoSE, each model configuration was trained, using 20 different random seeds for 2000 epochs.

Original: Each VICE model was initialized with two weight matrices, µ ∈ R m × d and log ( σ )

Original: ∈ R m × d , where m refers to the number of unique objects in the dataset (T HINGS : m = 1854 ;

Original: A DJECTIVES : m = 2372 ; F OOD : m = 36 ) and d , the initial dimensionality of the embedding, was set to 100 (the log ensures that σ is positive).

Original: In preliminary experiments, we observed that for sufficiently large d our dimensionality reduction method (see §5.2) prunes to similar representations, regardless of the choice of d .

Original: This is why we did not consider models with larger initial embedding dimensionality.

Original: Weight initialization We initialized the means of the variational distributions, µ , following a

Original: Kaiming He initialization [19].

Original: The logarithms of the scales of the variational distributions, log ( σ ) , were initialized with ϵ = − 1 /s µ , where s µ is the standard deviation over the entires of µ , so log( σ )

Original: =

Original: ϵ 1 .

Original: Hyperparameter grid The final grid was the Cartesian product of the following hyperparameter sets: π spike = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } , σ spike = { 0 . 125 , 0 .

Original: 5 , 1 . 0 , 2 . 0 } , σ slab = { 0 .

Original: 5 , 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 } , subject to the constraint σ spike ≪ σ slab , where combinations that did not satisfy the constraint were discarded.

Original: We observed that setting σ slab > 8 . 0 led to numerical overflow issues during optimization.

Original: For SPoSE, we used the same range as was done in Zheng et al.

Original: [50], with a finer grid of 64 values.

Original: Optimal hyperparameters We found the optimal VICE hyperparameter combination through a two step procedure.

Original: First, among the final 180 combinations (see Cartesian product above), we applied our pruning method (see §5.2) to each model and kept the subsets of dimensions where more than 5 objects had non-zero weight.

Original: For SPoSE we used the pruning heuristic proposed in Zheng et al.

Original: We defined the optimal hyperparameter combination as that with the lowest average cross-entropy error on the validation set across twenty different random initializations.

Original: The optimal hyperparameter combinations for VICE and SPoSE on the full datasets are reported in Table 2.

Original: [Table]

Original: [Table]

Original: [Table]

Original: Computational resources This study utilized the high-performance computational capabilities of the Biowulf Linux cluster at the National Institutes of Health, Bethesda, MD ( http://biowulf.nih.

Original: gov ) and the Raven and Cobra Linux clusters at the Max Planck Computing & Data Facility (MPCDF),

Original: Garching, Germany ( https://www.mpcdf.mpg.de/services/supercomputing/ ).

Original: The total number of CPU hours used were approximately 40 , 000 , 000 (Biowulf) and 5 , 000 (MPCDF).

Original: Accuracy upper bound For T HINGS [50, 21] and A DJECTIVES , a random subset of triplets was chosen to be presented multiple times to different participants.

Original: For a given triplet - repeated over many participants - this provides a way to estimate the distribution of responses over all participants.

Original: If for a given triplet the response distribution is (0 . 2 , 0 . 3 , 0 . 5) , then the best predictor for the participants’ responses is the third object.

Original: This results in an accuracy score of 50%, averaged across repetitions.

Original: Alternatively, one may observe a distribution of (0 . 1 , 0 . 8 , 0 . 1) for a different triplet.

Original: The best one could do

Original: is to identify the second object as the odd-one-out, and get 80% accuracy.

Original: From this, we can see that no classifier can do worse than 33%.

Original: Taking the average best prediction accuracy over all of the repeated triplets gives us an estimate for the best possible average prediction score.

Original: This is defined to be the upper bound for the prediction performance.

Original: F

Original: Generalization error bound

Original: F.1 Algorithm for generalization error upper bound

Original: Algorithm

Original: Input: µ

Original: M ← max( µ ) { ∆ 1 , . . . , ∆ m } ← { 0 . 05 , ..., 1 . 0 } ▷ pre-determined set of quantization scales α ← 0 .▷

Original: desired Type I error control rate δ ← αm for i ∈ { 1 , . . .

Original: , m } do µ † i :=

Original: quantize ( µ, ∆ i )

Original: ▷ quantization with ∆

Original: i ˆ R i

Original: =

Original: loss ( µ † i , D )

Original: ▷ (training) error ¯ R i :

Original: =

Original: ˆ R i + (cid:113) md log( ⌈ M/ ∆ i ⌉ +1)+log(1 /δ )

Original: 2 N train ▷ generalization upper bound end for i ∗ := arg min { ¯ R i , ..., ¯ R m }

Original: Output: ( µ † i ∗ , ¯ R i ∗ )

Original: ▷ ¯ R i ∗ holds with probability 1 − α

Original: We have the flexibility of choosing the quantization scale post-hoc.

Original: As long as we search over a pre-specified set of m quantization scales { ∆ 1 , . .

Original: . , ∆ m } , using a union bound, the PAC bound holds simultaneously for all quantized embeddings with probability at least 1 − mδ .

Original: Therefore, we can find the quantization scale that gives us the best probabilistic upper bound on generalization error.

Original: F.2 Quantization

Original: This section describes a number of empirical findings which support the feasibility of obtaining useful bounds by using our proposed quantization-based PAC bound (see §6) and the algorithm for obtaining retrospective generalization bounds (see Appendix F.1).

Original: Recall that we require two assumptions for our bounding approach to be effective.

Original: Specifically, we assume,

Original: Sparse embeddings obtained by either SPoSE or VICE can be quantized in a relatively coarse fashion, with only marginal losses in predictive performance.

Original: There exists an upper bound M on the largest value in an embedding.

Original: In the following we present empirical results which demonstrate that those assumptions are satisfied for the three datasets, T HINGS , A DJECTIVES , and F OOD , which we have used to evaluate VICE.

Original: Recall that it is possible to choose a certain quantization scale, ∆ , and round an embedding value to a non-negative integer multiple of ∆ .

Original: While we employ quantization mainly to use STL bounds for finite hypothesis classes, it could have benefits for interpretation as well (e.g., a dimension would consist of labels such as zero (0), very low (0.5), low (1), medium (1.5), high (2)).

Original: In Figure 6 we show generalization performances of VICE models as a function of the quantization scale, ∆ , for T HINGS , A DJECTIVES , and F OOD .

Original: To quantize the VICE embeddings, we used the same set of quantization scales for every dataset, ∆ ∈ { 0 . 05 , . .

Original: , 1 . 0 } , as defined in Algorithm 2.

Original: As ∆ increases, the number of bins, to which the embedding values can possibly be assigned, decreases.

Original: That is, the space of possible embedding values gets smaller as a function of increasing ∆ , and, therefore, maintaining generalization performance becomes more difficult.

Original: Note that quantization with ∆ = 0 is equivalent to performing inference with the original embeddings.

Original: As a result, the optimal generalization bounds that we get by using Algorithm 2 are typically obtained with ∆ ≤ 0 . 2 .

Original: However, we explore a much larger range since a priori we do not know the level of granularity needed to preserve most of the information in an embedding.

Original: Theoretically, there exist embeddings, such as highly sparse embeddings with nearly binary elements, which could maintain high accuracy at large quantization scales.

Original: Hence, the optimal level of quantization for the bound is an empirical question that may vary across datasets.

Original: However, we limit the upper range to 1, because if the largest weight is less than 2.7 (as we will examine below), then ∆ = 1 already reduces the number of distinct weight values to 3, which is small enough that we are likely to see diminishing returns by considering even larger discretization scales.

Original: Upper bound The upper bound, M , as defined in §6, for the VICE mean embeddings was approximately 2 . 7 for all datasets.

Original: Empirically, we found the maximum weights for T HINGS , A DJECTIVES , and F OOD to be 2 .

Original: 6285 , 2 . 2130 and 2 .

Original: 4307 , respectively, across all random initializations.

Original: Theoretically, this upper bound is not surprising due to a combination of two factors.

Original: First, in most datasets, the cross-entropy error goes to infinity as weights become arbitrarily large, due to the increasing over-confidence of incorrect probability estimates.

Original: Hence, the optimal cross-entropy is achieved with bounded weights.

Original: Second, almost any kind of regularization, including the ℓ 1 regularization used in SPoSE and the spike-and-slab regularization used for VICE, further encourages the weights to shrink.

Original: The spike-and-slab prior, in particular, penalizes large weights much higher than small weights, which is a desirable property in gradient-based optimization.

Original: G Dataset acquisition

Original: All datasets were collected by crowd-sourcing human responses to the triplet task described in §3 on the Amazon Mechanical Turk platform, using workers located in the United States.

Original: For A DJECTIVES , all words deemed offensive were removed from the list of adjectives being considered.

Original: There are no offensive images in either T HINGS or F OOD .

Original: All workers provided informed consent, and were compensated financially for their time ( 0 .

Original: 5 c per response, and additional 10 c per completed HIT).

Original: Workers could participate in blocks of 20 triplet trials and could choose to work on as many such blocks as they liked.

Original: The online research was approved by the Office of Human Research Subject

Original: Protection at the National Institutes of Health and conducted in accordance with all relevant ethical regulations.

Original: Worker ages were not assessed, and no personally identifiable information was collected.

Original: T HINGS and A DJECTIVES

Original: A total of 5,526 workers participated in collecting the first dataset (5,301 after exclusion; 3,159 female, 2,092 male, 19 other, 31 no response).

Original: A total of 336 workers participated in collecting the second dataset (325 after exclusion; 156 female, 103 male, 66 not reported).

Original: Workers were excluded if they exhibited overly fast responses in at least 5 sets of 20 trials (the speed cut-off was 25% or more responses < 800 ms and 50% or more responses < 1 , 100 ms) or if they carried out at least 200 trials and showed overly deterministic responses ( > 40% of responses in one of the three odd-one-out positions; expected value, 33% ).

Original: F OOD

Original: A total of 554 subjects participated in collecting the dataset (487 after exclusion).

Original: Workers were excluded if they exhibited overly fast responses (reaction time of less than 500ms).

Original: They were also excluded if they failed on either of two catch trials in each HIT, where they saw images of ‘+’, ‘-‘, ’-’ or ‘=’, ‘+’, or ‘=’ instead of food pictures; they were instructed on the slide to select the ‘+’.

Original: H

Original: Interpretability

Original: Here, we display the top 6 objects for each of the 45 VICE dimensions T HINGS [20].

Original: Objects were sorted in descending order according to their absolute embedding value.

Original: As we have done for every other experiment, we used the pruned median model to guarantee the extraction of a representative sample of embedding dimensions without being over-optimistic with respect to their interpretability (see §7.1 for how the median model was identified).