Original: C APTURING H UMAN C ATEGORY R EPRESENTATIONS BY S AMPLING IN D EEP F EATURE S PACES
Revised: C APTURING H UMAN C ATEGORY R EPRESENTATIONS BY S AMPLING IN D EEP F EATURE S PACES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Understanding how people represent categories is a core problem in cognitive science, with the ﬂexibility of human learning remaining a gold standard to which modern artiﬁcial intelligence and machine learning aspires.
Revised: Understanding how people represent categories is a core problem in cognitive science, with the ﬂexibility of human learning remaining a gold standard to which modern artiﬁcial intelligence and machine learning aspires.

Original: Decades of psychological research have yielded a variety of formal theories of categories, yet validating these theories with naturalistic stimuli remains a challenge.
Revised: Decades of psychological research have yielded a variety of formal theories of categories, yet validating these theories with naturalistic stimuli remains a challenge.

Original: The problem is that human category representations cannot be directly observed and running informative experiments with naturalistic stimuli such as images requires having a reasonable representation of these images.
Revised: The problem is that human category representations cannot be directly observed and running informative experiments with naturalistic stimuli such as images requires having a reasonable representation of these images.

Original: Deep neural networks have recently been successful in a range of computer vision tasks and provide a way to represent the features of images.
Revised: Deep neural networks have recently been successful in a range of computer vision tasks and provide a way to represent the features of images.

Original: In this paper, we outline a method for estimating the structure of human categories that draws on ideas from both cognitive science and machine learning, blending human-based algorithms with state-of-the-art deep representation learners.
Revised: In this paper, we outline a method for estimating the structure of human categories that draws on ideas from both cognitive science and machine learning, blending human-based algorithms with state-of-the-art deep representation learners.

Original: We provide qualitative and quantitative results as a proof of concept for the feasibility of the method.
Revised: We provide qualitative and quantitative results as a proof of concept for the feasibility of the method.

Original: Samples drawn from human distributions rival the quality of current state-of-the-art generative models and outperform alternative methods for estimating the structure of human categories.
Revised: Samples drawn from human distributions rival the quality of current state-of-the-art generative models and outperform alternative methods for estimating the structure of human categories.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Categorization (or classiﬁcation) is a central problem in cognitive science (Cohen & Lefebvre, 2005), artiﬁcial intelligence, and machine learning (Duda et al., 1973).
Revised: Categorization (or classiﬁcation) is a central problem in cognitive science (Cohen & Lefebvre, 2005), artiﬁcial intelligence, and machine learning (Duda et al., 1973).

Original: In its most general description, it concerns why and how we divide the world into discrete units (and various levels of abstraction), and what we do with this information.
Revised: In its most general description, it concerns why and how we divide the world into discrete units (and various levels of abstraction), and what we do with this information.

Original: The biggest challenge for studying human categorization is that the content of mental category representations cannot be directly observed, which has led to development of laboratory methods for estimating this content from human behavior.
Revised: The biggest challenge for studying human categorization is that the content of mental category representations cannot be directly observed, which has led to development of laboratory methods for estimating this content from human behavior.

Original: Because these methods rely on small artiﬁcial stimulus sets with handcrafted or low-dimensional feature sets, they are ill-suited to the study of categorization as an intelligent process, which is principally motivated by people’s robust categorization performance in complex ecological settings.
Revised: Because these methods rely on small artiﬁcial stimulus sets with handcrafted or low-dimensional feature sets, they are ill-suited to the study of categorization as an intelligent process, which is principally motivated by people’s robust categorization performance in complex ecological settings.

Original: One of the challenges of applying psychological methods to realistic stimuli such as natural images is ﬁnding a way to represent them.
Revised: One of the challenges of applying psychological methods to realistic stimuli such as natural images is ﬁnding a way to represent them.

Original: Recent work in machine learning has shown that deep learning models, such as convolutional neural networks, perform well on a range of computer vision tasks (LeCun et al., 2015).
Revised: Recent work in machine learning has shown that deep learning models, such as convolutional neural networks, perform well on a range of computer vision tasks (LeCun et al., 2015).

Original: The features discovered by these models provide a way to represent complex images compactly.
Revised: The features discovered by these models provide a way to represent complex images compactly.

Original: It may be possible to express human category structure using these features, an idea supported by recent work in cognitive science (Lake et al., 2015; Peterson et al., 2016).
Revised: It may be possible to express human category structure using these features, an idea supported by recent work in cognitive science (Lake et al., 2015; Peterson et al., 2016).

Original: Ideally, experimental methods could be combined with state-of-the-art deep learning models to estimate the structure of human categories with as few assumptions as possible and while avoiding the problem of dataset bias.
Revised: Ideally, experimental methods could be combined with state-of-the-art deep learning models to estimate the structure of human categories with as few assumptions as possible and while avoiding the problem of dataset bias.

Original: In what follows, we propose a method that uses a human in the loop to directly estimate arbitrary distributions over complex feature spaces, adapting a framework that can exploit advances in deep architectures and computing power to increasingly capture and sharpen the precise structure of human category representations.
Revised: In what follows, we propose a method that uses a human in the loop to directly estimate arbitrary distributions over complex feature spaces, adapting a framework that can exploit advances in deep architectures and computing power to increasingly capture and sharpen the precise structure of human category representations.

Original: Such knowledge is crucial to forming an ecological theory of intelligent categorization behavior and to providing a ground-truth benchmark to guide and inspire future work in machine learning.
Revised: Such knowledge is crucial to forming an ecological theory of intelligent categorization behavior and to providing a ground-truth benchmark to guide and inspire future work in machine learning.

Original:  B ACKGROUND 2. E STIMATING THE STRUCTURE OF HUMAN CATEGORIES
Revised:  B ACKGROUND 2. E STIMATING THE STRUCTURE OF HUMAN CATEGORIES

Original: Methods for estimating human category templates have existed for some time.
Revised: Methods for estimating human category templates have existed for some time.

Original: In psychophysics, the most popular and well-understood method is known as classiﬁcation images (Ahumada, 1996).
Revised: In psychophysics, the most popular and well-understood method is known as classiﬁcation images (Ahumada, 1996).

Original: In this experimental procedure, a human participant is shown stimuli from two classes, A and B, each with white noise overlaid, and asked to indicate the correct label.
Revised: In this experimental procedure, a human participant is shown stimuli from two classes, A and B, each with white noise overlaid, and asked to indicate the correct label.

Original: On most trials, the participant will select the exemplar generated from the category in question.
Revised: On most trials, the participant will select the exemplar generated from the category in question.

Original: However, if the added white noise signiﬁcantly perturbs features of the image important to making that distinction, they may fail.
Revised: However, if the added white noise signiﬁcantly perturbs features of the image important to making that distinction, they may fail.

Original: Exploiting this, we can estimate the decision boundary from a number of these trials using the simple formula:
Revised: Exploiting this, we can estimate the decision boundary from a number of these trials using the simple formula:

Original: [Equation]
Revised: [Equation]

Original: where n XY is the average of the noise across trials where the correct class is X
Revised: where n XY is the average of the noise across trials where the correct class is X

Original: but the observer chooses Y .
Revised: but the observer chooses Y .

Original: Vondrick et al.
Revised: Vondrick et al.

Original: (2015) used a variation on classiﬁcation images using invertible deep feature spaces.
Revised: (2015) used a variation on classiﬁcation images using invertible deep feature spaces.

Original: In order to avoid dataset bias introduced through perturbing real class exemplars, white noise in the feature space was used to generate stimuli.
Revised: In order to avoid dataset bias introduced through perturbing real class exemplars, white noise in the feature space was used to generate stimuli.

Original: In this special case, category templates reduce to n A − n B .
Revised: In this special case, category templates reduce to n A − n B .

Original: In the experiment, on each trial participants were asked to select which of two images (inverted from feature noise) most resembled a particular category.
Revised: In the experiment, on each trial participants were asked to select which of two images (inverted from feature noise) most resembled a particular category.

Original: Because the feature vectors were random, thousands of images could be pre-inverted ofﬂine using methods that require access to large data sets.
Revised: Because the feature vectors were random, thousands of images could be pre-inverted ofﬂine using methods that require access to large data sets.

Original: This early inversion method was applied to mean feature vectors for thousands of positive choices in the experiments and yielded qualitatively decipherable category template images, as well as better machine classiﬁcation decision boundaries that were regularized by human bias.
Revised: This early inversion method was applied to mean feature vectors for thousands of positive choices in the experiments and yielded qualitatively decipherable category template images, as well as better machine classiﬁcation decision boundaries that were regularized by human bias.

Original: Under the assumption that human category distributions are Gaussian with equal variance, this method yields the vector that aligns with the line between means, although a massive number of pairs of random vectors (trials) are required.
Revised: Under the assumption that human category distributions are Gaussian with equal variance, this method yields the vector that aligns with the line between means, although a massive number of pairs of random vectors (trials) are required.

Original: Greene et al.
Revised: Greene et al.

Original: (2014) conducted similar experiments in which a low-dimensional multi-scale gabor
Revised: (2014) conducted similar experiments in which a low-dimensional multi-scale gabor

Original: PCA basis was used to represent black-and-white images of scenes.
Revised: PCA basis was used to represent black-and-white images of scenes.

Original: Participants indicated which of two images was more similar to a seen image or a mental image of a scene.
Revised: Participants indicated which of two images was more similar to a seen image or a mental image of a scene.

Original: These judgments were used in an online genetic algorithm to lead participants to converge to their mental image.
Revised: These judgments were used in an online genetic algorithm to lead participants to converge to their mental image.

Original: This method allowed for fast, direct estimation of mental templates, although it lacks even the few theoretical guarantees of the Vondrick et al. (2015) method and yields only a single meaningful image.
Revised: This method allowed for fast, direct estimation of mental templates, although it lacks even the few theoretical guarantees of the Vondrick et al. (2015) method and yields only a single meaningful image.

Original: Finally, an alternative to classiﬁcation images, Markov Chain Monte Carlo with People (MCMCP;
Revised: Finally, an alternative to classiﬁcation images, Markov Chain Monte Carlo with People (MCMCP;

Original: Sanborn and Grifﬁths, 2007), constructs an experimental procedure by which humans act as a valid acceptance function in the Metropolis–Hastings algorithm, exploiting the fact that Luce’s choice axiom, a well-known model of human choice behavior, is equivalent to the Barker acceptance function (see equation in Figure 1).
Revised: Sanborn and Grifﬁths, 2007), constructs an experimental procedure by which humans act as a valid acceptance function in the Metropolis–Hastings algorithm, exploiting the fact that Luce’s choice axiom, a well-known model of human choice behavior, is equivalent to the Barker acceptance function (see equation in Figure 1).

Original: On the ﬁrst trial, a stimulus is drawn arbitrarily from the parameter space and compared to a new proposed stimulus nearby to it in the parameter space.
Revised: On the ﬁrst trial, a stimulus is drawn arbitrarily from the parameter space and compared to a new proposed stimulus nearby to it in the parameter space.

Original: The participant makes a forced choice of the better exemplar of the category.
Revised: The participant makes a forced choice of the better exemplar of the category.

Original: If the initial stimulus is chosen, the Markov chain remains in that state.
Revised: If the initial stimulus is chosen, the Markov chain remains in that state.

Original: If the proposed stimulus is chosen, the chain moves to the proposed state.
Revised: If the proposed stimulus is chosen, the chain moves to the proposed state.

Original: The process repeats.
Revised: The process repeats.

Original: MCMCP has been successfully employed to capture a number of different mental categories (Sanborn & Grifﬁths, 2007; Martin et al., 2012), and though these spaces are higher-dimensional than those in previous laboratory experiments, they are still relatively small and artiﬁcial compared to real images.
Revised: MCMCP has been successfully employed to capture a number of different mental categories (Sanborn & Grifﬁths, 2007; Martin et al., 2012), and though these spaces are higher-dimensional than those in previous laboratory experiments, they are still relatively small and artiﬁcial compared to real images.

Original: Unlike classiﬁcation images, this method makes no assumptions about the structure of the category distributions and thus can estimate means, variances, and higher order moments.
Revised: Unlike classiﬁcation images, this method makes no assumptions about the structure of the category distributions and thus can estimate means, variances, and higher order moments.

Original: Therefore we take it as a basic starting point for this paper.
Revised: Therefore we take it as a basic starting point for this paper.

Original: 2. D EEP N EURAL N ETWORKS FOR I MAGES
Revised: 2. D EEP N EURAL N ETWORKS FOR I MAGES

Original: Deep convolutional neural networks (CNN; LeCun et al., 1989) like AlexNet (Krizhevsky et al., 2012) are excellent for high-accuracy natural image classiﬁcation, and learn feature and category representations that are generally useful for a number of perceptual tasks.
Revised: Deep convolutional neural networks (CNN; LeCun et al., 1989) like AlexNet (Krizhevsky et al., 2012) are excellent for high-accuracy natural image classiﬁcation, and learn feature and category representations that are generally useful for a number of perceptual tasks.

Original: However, they can be difﬁcult to interpret, especially in how they relate to human categorization behavior.
Revised: However, they can be difﬁcult to interpret, especially in how they relate to human categorization behavior.

Original: [Figure]
Revised: [Figure]

Original: Generative Adversarial Networks (
Revised: Generative Adversarial Networks (

Original: Goodfellow
Revised: Goodfellow

Original: ) and Variational Autoencoders (VAEs; Kingma & Welling, 2014) provide a generative approach to understanding image categories.
Revised: ) and Variational Autoencoders (VAEs; Kingma & Welling, 2014) provide a generative approach to understanding image categories.

Original: In particular, these generative models provide an opportunity to examine how inference over a set of images and categories is performed.
Revised: In particular, these generative models provide an opportunity to examine how inference over a set of images and categories is performed.

Original: Generative models frame their approach theoretically on a Bayesian decomposition of image-label joint density, p ( z, x )
Revised: Generative models frame their approach theoretically on a Bayesian decomposition of image-label joint density, p ( z, x )

Original: = p ( z ) p ( x | z ) , where z and x are random variables that represent a natural image distribution and its feature representation.
Revised: = p ( z ) p ( x | z ) , where z and x are random variables that represent a natural image distribution and its feature representation.

Original: Therfore p ( z ) represents the distribution of natural images, and p ( x | z ) is the distribution of feature vectors given an image.
Revised: Therfore p ( z ) represents the distribution of natural images, and p ( x | z ) is the distribution of feature vectors given an image.

Original: Together, p ( z ) and p ( x | z ) produce a high-dimensional constraint on image generation.
Revised: Together, p ( z ) and p ( x | z ) produce a high-dimensional constraint on image generation.

Original: This allows for easy visualization of the learned latent space, a property we will exploit presently.
Revised: This allows for easy visualization of the learned latent space, a property we will exploit presently.

Original:  M ARKOV C HAIN M ONTE C ARLO IN D EEP F EATURE S PACES
Revised:  M ARKOV C HAIN M ONTE C ARLO IN D EEP F EATURE S PACES

Original: In the typical MCMCP experiment, as explained above, the participant judges pairs of stimuli, and the chosen stimuli serve as samples from the category distribution of interest.
Revised: In the typical MCMCP experiment, as explained above, the participant judges pairs of stimuli, and the chosen stimuli serve as samples from the category distribution of interest.

Original: This method is effective as long as noise can be added to dimensions in the stimulus parameter space to create meaningful changes in content.
Revised: This method is effective as long as noise can be added to dimensions in the stimulus parameter space to create meaningful changes in content.

Original: When viewing natural images for example, in the space of all pixel intensities, added noise is very unlikely to modify the image in interesting ways.
Revised: When viewing natural images for example, in the space of all pixel intensities, added noise is very unlikely to modify the image in interesting ways.

Original: We propose to instead perturb images in a deep feature space that captures only essential variation.
Revised: We propose to instead perturb images in a deep feature space that captures only essential variation.

Original: We can then show participants images decoded from these feature representations in order to relate human judgments to the underlying latent space of interest, where category distributions are easier to learn.
Revised: We can then show participants images decoded from these feature representations in order to relate human judgments to the underlying latent space of interest, where category distributions are easier to learn.

Original: The resulting judgments (samples) approximate distributions that both derive arbitrary human category boundaries for natural images and can be sampled from to create new images, yielding new human-like generative image models.
Revised: The resulting judgments (samples) approximate distributions that both derive arbitrary human category boundaries for natural images and can be sampled from to create new images, yielding new human-like generative image models.

Original: A schematic of this entire procedure is illustrated in Figure 1.
Revised: A schematic of this entire procedure is illustrated in Figure 1.

Revised: Note that the method we propose doesn’t actually require an inference network in practice, since we always know which latent code produced the images shown to the subjects.

Revised: This is true as long as we initialize our MCMC chain states as random points in the latent space as opposed to pixel space.

Original: More speciﬁcally, popular deep generative networks such as GANs and VAEs learn a probability distribution p ( z ) over some latent variable z , from which the objective of the network encourages of the mapping (generator or decoder)
Revised: More speciﬁcally, popular deep generative networks such as GANs and VAEs learn a probability distribution p ( z ) over some latent variable z , from which the objective of the network encourages of the mapping (generator or decoder)

Original: M that p ( f ( z ; M ))
Revised: M that p ( f ( z ; M ))

Original: = p ( x ) , where p ( x ) is the distribution over pixels in the dataset, f is the family of deterministic functions s.t.
Revised: = p ( x ) , where p ( x ) is the distribution over pixels in the dataset, f is the family of deterministic functions s.t.

Original: f
Revised: f

Original: : M × z → x .
Revised: : M × z → x .

Original: If we assume that human psychological feature space y results from some invertible transformation T of x (it is simply some learned transformation of raw input), then correspondingly p ( g ( y ; T − 1 ) = p ( x ) , where g is the family of functions s.t.
Revised: If we assume that human psychological feature space y results from some invertible transformation T of x (it is simply some learned transformation of raw input), then correspondingly p ( g ( y ; T − 1 ) = p ( x ) , where g is the family of functions s.t.

Original: g : T − 1 × y
Revised: g : T − 1 × y

Original: → x . Because people in our experiments see only actual images x , perceived as y , the judgments they make are related to the deep latent space through the following equation:
Revised: → x . Because people in our experiments see only actual images x , perceived as y , the judgments they make are related to the deep latent space through the following equation:

Original: [Equation]
Revised: [Equation]

Original: Equation 2 assumes that humans approximate p ( x ) analogously to an unsupervised learner, however, even if T is a more speciﬁc feature representation supervised by classiﬁcation pressure in the environment, we can assume information is only lost and not added, so that our deep generative models, if successful in their goal, will ultimately encode this information as a subset.
Revised: Equation 2 assumes that humans approximate p ( x ) analogously to an unsupervised learner, however, even if T is a more speciﬁc feature representation supervised by classiﬁcation pressure in the environment, we can assume information is only lost and not added, so that our deep generative models, if successful in their goal, will ultimately encode this information as a subset.

Original: Further, the equation above entails that p ( y ) need not be identical to p ( z ) , and M need not equal T − 1 .
Revised: Further, the equation above entails that p ( y ) need not be identical to p ( z ) , and M need not equal T − 1 .

Original: However, for the practical sake of convergence using human participants, we would prefer that psychological space y be similar to the deep latent space z — we hope they contain the same relevant featural content for most images.
Revised: However, for the practical sake of convergence using human participants, we would prefer that psychological space y be similar to the deep latent space z — we hope they contain the same relevant featural content for most images.

Original: While this assumption is hard to verify beyond face validity at this time, recent work suggests that some deep features spaces can be linearly transformed to approximate human psychological space (Peterson et al., 2016), and so we assume the deep latent space is relevant enough to human feature representations to act as surrogate.
Revised: While this assumption is hard to verify beyond face validity at this time, recent work suggests that some deep features spaces can be linearly transformed to approximate human psychological space (Peterson et al., 2016), and so we assume the deep latent space is relevant enough to human feature representations to act as surrogate.

Original: There are several theoretical advantages to our method over previous efforts.
Revised: There are several theoretical advantages to our method over previous efforts.

Original: First, MCMCP can capture arbitrary distributions, so it is not as sensitive to the structure of the underlying low-dimensional feature space and should provide better category boundaries than classiﬁcation images when required.
Revised: First, MCMCP can capture arbitrary distributions, so it is not as sensitive to the structure of the underlying low-dimensional feature space and should provide better category boundaries than classiﬁcation images when required.

Original: This is important when using various deep features spaces that were learned with different constraints.
Revised: This is important when using various deep features spaces that were learned with different constraints.

Original: MCMC inherently spends less time in low probability regions and should in theory waste fewer trials.
Revised: MCMC inherently spends less time in low probability regions and should in theory waste fewer trials.

Original: Having generated the images online and as a function of the participant’s decisions, there is no data set or sampling bias, and auto-correlation can be addressed by removing temporally adjacent samples from the chain.
Revised: Having generated the images online and as a function of the participant’s decisions, there is no data set or sampling bias, and auto-correlation can be addressed by removing temporally adjacent samples from the chain.

Revised: Finally, using a deep generator provides drastically clearer samples than shallow reconstruction methods, and can be potentially be trained end-to-end with an inference network that allows us to categorize new images using the distribution learned.

Original:  H UMAN E XPERIMENTS
Revised:  H UMAN E XPERIMENTS

Original: For our experiments, we explored two image generator networks trained on various datasets.
Revised: For our experiments, we explored two image generator networks trained on various datasets.

Original: Since even relatively low-dimensional deep image embeddings are large compared to controlled laboratory stimulus parameter spaces, we use a hybrid proposal distribution in which a Gaussian with a low variance is used with probability P and a Gaussian with a high variance is used with probability 1 − P .
Revised: Since even relatively low-dimensional deep image embeddings are large compared to controlled laboratory stimulus parameter spaces, we use a hybrid proposal distribution in which a Gaussian with a low variance is used with probability P and a Gaussian with a high variance is used with probability 1 − P .

Original: This allows participants to both reﬁne and escape nearby modes, but is simple enough to avoid excessive experimental piloting that more advanced proposal methods often require.
Revised: This allows participants to both reﬁne and escape nearby modes, but is simple enough to avoid excessive experimental piloting that more advanced proposal methods often require.

Original: Participants in all experiments completed exactly 64 trials (image comparisons), collectively taking about 5 minutes, containing segments of several chains for multiple categories.
Revised: Participants in all experiments completed exactly 64 trials (image comparisons), collectively taking about 5 minutes, containing segments of several chains for multiple categories.

Original: The order of the categories and chains within those categories were always interleaved.
Revised: The order of the categories and chains within those categories were always interleaved.

Original: All experiments were conducted on Amazon Mechanical Turk.
Revised: All experiments were conducted on Amazon Mechanical Turk.

Original: 4. A F IRST P ROOF OF C ONCEPT : F ACE C ATEGORIES 4.1. M ETHODS
Revised: 4. A F IRST P ROOF OF C ONCEPT : F ACE C ATEGORIES 4.1. M ETHODS

Original: We ﬁrst test our method using DCGAN (Radford et al., 2015) trained on the Asian Faces Dataset.
Revised: We ﬁrst test our method using DCGAN (Radford et al., 2015) trained on the Asian Faces Dataset.

Original: We chose this dataset because it requires a deep architecture to produce reasonable samples (unlike
Revised: We chose this dataset because it requires a deep architecture to produce reasonable samples (unlike

Original: MNIST for example), yet it is constrained enough to test-drive our method using a relatively simple latent space.
Revised: MNIST for example), yet it is constrained enough to test-drive our method using a relatively simple latent space.

Original: Four chains for each of four categories (male, female, happy, and sad) were used.
Revised: Four chains for each of four categories (male, female, happy, and sad) were used.

Original: Proposals were generated from an isometric Gaussian with an SD of 0.25 50% of the time, and  2 otherwise.
Revised: Proposals were generated from an isometric Gaussian with an SD of 0.25 50% of the time, and [Figure] 2 otherwise.

Original: In addition, we conducted a baseline in which two new initial state proposals were drawn on every trial, and were independent of previous trials (classiﬁcation images).
Revised: In addition, we conducted a baseline in which two new initial state proposals were drawn on every trial, and were independent of previous trials (classiﬁcation images).

Original: The ﬁnal dataset contained 50 participants and over 3 , 200 trials (samples) in total for all chains.
Revised: The ﬁnal dataset contained 50 participants and over 3 , 200 trials (samples) in total for all chains.

Original: The baseline classiﬁcation images (CI) dataset contained the same number of trials and participants.[Figure]
Revised: The baseline classiﬁcation images (CI) dataset contained the same number of trials and participants.

Original: 4.1. R ESULTS
Revised: 4.1. R ESULTS

Original: MCMCP chains are visualized using Fisher Linear Discriminant Analysis in Figure 2, along with the resulting averages for each chain and each category.
Revised: MCMCP chains are visualized using Fisher Linear Discriminant Analysis in Figure 2, along with the resulting averages for each chain and each category.

Original: Chain means within a category show interesting variation, yet converge to very similar regions in the latent space.
Revised: Chain means within a category show interesting variation, yet converge to very similar regions in the latent space.

Original: Interestingly, it appears that female faces are viewed as more happy (although signiﬁcantly less than the happy category means), whereas male faces are more neutral.
Revised: Interestingly, it appears that female faces are viewed as more happy (although signiﬁcantly less than the happy category means), whereas male faces are more neutral.

Original: This effect that could be due to dataset used for training, especially if the women pictured smile more in photos.
Revised: This effect that could be due to dataset used for training, especially if the women pictured smile more in photos.

Original: What’s more important however is the apparent improvement over the baseline CI means shown in the right-most column of Figure 2.
Revised: What’s more important however is the apparent improvement over the baseline CI means shown in the right-most column of Figure 2.

Original: MCMCP means appear to have converged quickly, whereas CI means only show a moderate resemblance to their corresponding category.
Revised: MCMCP means appear to have converged quickly, whereas CI means only show a moderate resemblance to their corresponding category.

Original: All four CI means appear closer to a mean face, which is what one would expect from averages of noise.
Revised: All four CI means appear closer to a mean face, which is what one would expect from averages of noise.

Original: We validate this improvement with a human experiment in which 30 participants made forced choices between CI and MCMCP means.
Revised: We validate this improvement with a human experiment in which 30 participants made forced choices between CI and MCMCP means.

Original: The results are reported in Figure 3.
Revised: The results are reported in Figure 3.

Original: MCMCP means are consistently highly preferred as representations of each category as compared to CI.
Revised: MCMCP means are consistently highly preferred as representations of each category as compared to CI.

Original: This remained true even when an additional 50 participants (total of 100 ) were run on the CI task, obtaining twice as many image comparison trials as MCMCP.
Revised: This remained true even when an additional 50 participants (total of 100 ) were run on the CI task, obtaining twice as many image comparison trials as MCMCP.

Original: 4. L ARGER N ETWORKS & L ARGER S PACES
Revised: 4. L ARGER N ETWORKS & L ARGER S PACES

Original: The results in the previous section show that reasonable category templates can be obtained using our method, yet the complexity of the stimulus space used does not rival that of large object classiﬁcation networks.
Revised: The results in the previous section show that reasonable category templates can be obtained using our method, yet the complexity of the stimulus space used does not rival that of large object classiﬁcation networks.

Original: In this section, we tackle a more challenging (and interesting) form of the problem.
Revised: In this section, we tackle a more challenging (and interesting) form of the problem.

Original: To do this, we employ a bidirectional generative adversarial network (BiGAN; Donahue et al., 2016) trained on the entire 1.2 million-image ilsvrc12 dataset ( 64 × 64 center-cropped).
Revised: To do this, we employ a bidirectional generative adversarial network (BiGAN; Donahue et al., 2016) trained on the entire 1.2 million-image ilsvrc12 dataset ( 64 × 64 center-cropped).

Original: We also simultaneously learn a corresponding inference network, which regularizes the rest of the network and produces unconditional samples competitive with the state-of-the-art.
Revised: We also simultaneously learn a corresponding inference network, which regularizes the rest of the network and produces unconditional samples competitive with the state-of-the-art.

Original: This also allows for the later possibility of comparing human distributions with other networks as well as assessing machine classiﬁcation performance with new images based on the granular human biases captured.
Revised: This also allows for the later possibility of comparing human distributions with other networks as well as assessing machine classiﬁcation performance with new images based on the granular human biases captured.

Original: 4.2. M ETHODS
Revised: 4.2. M ETHODS

Original: Our generator network was trained given uniform rather than Gaussian noise, which allows us to guarantee participants cannot get lost in extremely improbable regions.
Revised: Our generator network was trained given uniform rather than Gaussian noise, which allows us to guarantee participants cannot get lost in extremely improbable regions.

Original: Additionally, we avoid proposing states outside of this hypercube by forcing z to wrap around (proposals that travel outside of z are injected back in from the opposite direction by the amount originally exceeded).
Revised: Additionally, we avoid proposing states outside of this hypercube by forcing z to wrap around (proposals that travel outside of z are injected back in from the opposite direction by the amount originally exceeded).

Original: In particular we run our MCMC chains through an unbounded state space by redeﬁning each bounded dimensionz k as
Revised: In particular we run our MCMC chains through an unbounded state space by redeﬁning each bounded dimension

Original: [Equation]
Revised: [Equation]

Original: Proposals were generated from an isometric Gaussian with an SD of 0 .
Revised: Proposals were generated from an isometric Gaussian with an SD of 0 .

Original: 1 60% of the time, and 0 .otherwise.
Revised: 1 60% of the time, and 0 .otherwise.

Original: We use this network to obtain large chains for two groups of ﬁve categories.
Revised: We use this network to obtain large chains for two groups of ﬁve categories.

Original: Group1 included bottle , car , ﬁre hydrant , and person , television , following Vondrick et al.
Revised: Group1 included bottle , car , ﬁre hydrant , and person , television , following Vondrick et al.

Original: Group2 included bird , body of water , ﬁsh , ﬂower , and landscape .
Revised: Group2 included bird , body of water , ﬁsh , ﬂower , and landscape .

Original: Each chain was approximately 1 , 040 states long, and four of these chains were used for each category (approximately 4 , 160 ).
Revised: Each chain was approximately 1 , 040 states long, and four of these chains were used for each category (approximately 4 , 160 ).

Original: In total, across both groups of categories, we obtained exactly 41 , 600 samples from 650 participants.
Revised: In total, across both groups of categories, we obtained exactly 41 , 600 samples from 650 participants.

Original: To demonstrate the efﬁciency and ﬂexibility of our method compared to alternatives, we obtained an equivalent number of trials for all categories using the variant of classiﬁcation images introduced in Vondrick et al.
Revised: To demonstrate the efﬁciency and ﬂexibility of our method compared to alternatives, we obtained an equivalent number of trials for all categories using the variant of classiﬁcation images introduced in Vondrick et al.

Original: (2015), with the exception that we used our BiGAN generator instead of ofﬂine inversion previously used.
Revised: (2015), with the exception that we used our BiGAN generator instead of ofﬂine inversion previously used.

Original: This also serves as an important baseline (since it estimates the simplest possible template) against which to quantitatively evaluate our method.
Revised: This also serves as an important baseline (since it estimates the simplest possible template) against which to quantitatively evaluate our method.

Original: 4.2. R ESULTS
Revised: 4.2. R ESULTS

Original: The acceptance rate was approximately 50% for both category groups, which is near the common goal for MCMCP experiments.
Revised: The acceptance rate was approximately 50% for both category groups, which is near the common goal for MCMCP experiments.

Original: The samples for all ten categories are shown in Figure 5B and D using Fisher Linear Discriminant Analysis.
Revised: The samples for all ten categories are shown in Figure 5B and D using Fisher Linear Discriminant Analysis.

Original: Similar to the face chains, the four chains for each category converge to similar regions in space, largely away from other categories.
Revised: Similar to the face chains, the four chains for each category converge to similar regions in space, largely away from other categories.

Original: In contrast, classiﬁcation images shows little separation with so few trials (5C and D).
Revised: In contrast, classiﬁcation images shows little separation with so few trials (5C and D).

Original: Previous work suggests that at least an order of magnitude higher number of comparisons may be needed for satisfactory estimation of category means.
Revised: Previous work suggests that at least an order of magnitude higher number of comparisons may be needed for satisfactory estimation of category means.

Original: Our method estimates well-separated category means in a manageable number of trials, allowing for the method to scale greatly.
Revised: Our method estimates well-separated category means in a manageable number of trials, allowing for the method to scale greatly.

Original: This makes sense given that it proposes comparisons  between arbitrary images, potentially wasting many trials, and clearly suffers from a great deal of noise.
Revised: This makes sense given that it proposes comparisons [Table] between arbitrary images, potentially wasting many trials, and clearly suffers from a great deal of noise.

Original: Beyond yielding our decision rule, our method additionally produces a density estimate of the entire category distribution.
Revised: Beyond yielding our decision rule [Table] our method additionally produces a density estimate of the entire category distribution.

Revised: In classiﬁcation images

Original: In classiﬁcation images, mean template images can be viewed, while we are able to visualize several modes in the category distribution.
Revised: [Table] mean template images can be viewed [Table] while we are able to visualize several modes in the category distribution.

Original: Figure 4 visualizes these modes using the means of each component in a mixture of Gaussians density estimate.
Revised: Figure 4 visualizes these modes using the means of each component in a mixture of Gaussians density estimate.

Original: This produces[Table] realistic-looking multi-modal mental category templates, which to our knowledge has never been accomplished with respect to natural image categories.
Revised: This produces realistic-looking multi-modal mental category templates, which to our knowledge has never been accomplished with respect to natural image categories.

Original: 4. U SING E STIMATED H UMAN C ATEGORIES FOR C LASSIFICATION
Revised: 4. U SING E STIMATED H UMAN C ATEGORIES FOR C LASSIFICATION

Original: We also provide a quantitative assessment of the samples we obtained and compare them to classiﬁcation images (CI) using an external classiﬁcation task.
Revised: We also provide a quantitative assessment of the samples we obtained and compare them to classiﬁcation images (CI) using an external classiﬁcation task.

Original: To do this, we scraped approximately 500 images from Flickr for each of our ten categories [Table] which was used for a classiﬁcation task.
Revised: To do this, we scraped approximately 500 images from Flickr for each of our ten categories, which was used for a classiﬁcation task.

Original: To classify the images with using our samples, we used (1) the nearest-mean decision rule [Table] and (2) a decision rule based on the highest log-probability given by our ten density estimates.
Revised: To classify the images with using our samples, we used (1) the nearest-mean decision rule, and (2) a decision rule based on the highest log-probability given by our ten density estimates.

Original: For, classiﬁcation images, only a nearest-mean decision rule can be tested.
Revised: For, classiﬁcation images, only a nearest-mean decision rule can be tested.

Original: In all cases [Table] decision rules based on our MCMCP-obtained samples overall outperform a nearestmean decision rule using classiﬁcation images (see Figure 1).
Revised: In all cases, decision rules based on our MCMCP-obtained samples overall outperform a nearestmean decision rule using classiﬁcation images (see Figure 1).

Original: In category group 1, the MCMCP density performed best and was more even across classes.
Revised: In category group 1, the MCMCP density performed best and was more even across classes.

Original: In category group 2, nearest-mean using our MCMCP samples did much better than a density estimate or CI-based nearest-mean.
Revised: In category group 2, nearest-mean using our MCMCP samples did much better than a density estimate or CI-based nearest-mean.

Original:  D ISCUSSION & C ONCLUSION
Revised:  D ISCUSSION & C ONCLUSION

Original: Our results demonstrate the potential of our method, which leverages both psychological methods and deep surrogate representations to make the problem of capturing human category representations tractable.
Revised: Our results demonstrate the potential of our method, which leverages both psychological methods and deep surrogate representations to make the problem of capturing human category representations tractable.

Original: The ﬂexibility of our method in ﬁtting arbitrary generative models allows us to visualize multi-modal category templates for the ﬁrst time, and improve on human-based classiﬁcation performance benchmarks.
Revised: The ﬂexibility of our method in ﬁtting arbitrary generative models allows us to visualize multi-modal category templates for the ﬁrst time, and improve on human-based classiﬁcation performance benchmarks.

Original: It is difﬁcult to guarantee that our chains explored enough of the relevant space to actually capture the concepts in their entirety, but the diversity in the modes visualized and the improvement in class separation achieved are positive indications that we are on the right track.
Revised: It is difﬁcult to guarantee that our chains explored enough of the relevant space to actually capture the concepts in their entirety, but the diversity in the modes visualized and the improvement in class separation achieved are positive indications that we are on the right track.

Original: Further, the framework we present can be straightforwardly improved as generative image models advance, and a number of known methods for improving the speed, reach, and accuracy of MCMC algorithms can be applied to MCMCP make better use of costly human trials.
Revised: Further, the framework we present can be straightforwardly improved as generative image models advance, and a number of known methods for improving the speed, reach, and accuracy of MCMC algorithms can be applied to MCMCP make better use of costly human trials.

Original: There are several obvious limitations of our method.
Revised: There are several obvious limitations of our method.

Original: First, the structure of the underlying feature spaces used may either lack the expressiveness (some features may be missing) or the constraints (too many irrelevant features or possible images wastes too many trials) needed to map all characteristics of human mental categories in a practical number of trials.
Revised: First, the structure of the underlying feature spaces used may either lack the expressiveness (some features may be missing) or the constraints (too many irrelevant features or possible images wastes too many trials) needed to map all characteristics of human mental categories in a practical number of trials.

Original: Even well-behaved spaces are very large and require many trials to reach convergence.
Revised: Even well-behaved spaces are very large and require many trials to reach convergence.

Original: Addressing this will require continuing exploration of a variety of generative image models.
Revised: Addressing this will require continuing exploration of a variety of generative image models.

Original: We see our work are as part of an iterative reﬁnement process that can yield more granular human observations and inform new deep network objectives and architectures, both of which may yet converge on a proper, yet tractable model of real-world human categorization.
Revised: We see our work are as part of an iterative reﬁnement process that can yield more granular human observations and inform new deep network objectives and architectures, both of which may yet converge on a proper, yet tractable model of real-world human categorization.