Original: CIGMO: L EARNING CATEGORICAL INVARIANT DEEP GENERATIVE MODELS FROM GROUPED DATA
Revised: CIGMO: L EARNING CATEGORICAL INVARIANT DEEP GENERATIVE MODELS FROM GROUPED DATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Images of general objects are often composed of three hidden factors: category (e.g., car or chair), shape (e.g., particular car form), and view (e.g., 3D orientation).
Revised: Images of general objects are often composed of three hidden factors: category (e.g., car or chair), shape (e.g., particular car form), and view (e.g., 3D orientation).

Original: While there have been many disentangling models that can discover either a category or shape factor separately from a view factor, such models typically cannot capture the structure of general objects that the diversity of shapes is much larger across categories than within a category.
Revised: While there have been many disentangling models that can discover either a category or shape factor separately from a view factor, such models typically cannot capture the structure of general objects that the diversity of shapes is much larger across categories than within a category.

Original: Here, we propose a novel generative model called CIGMO, which can learn to represent the category, shape, and view factors at once only with weak supervision.
Revised: Here, we propose a novel generative model called CIGMO, which can learn to represent the category, shape, and view factors at once only with weak supervision.

Original: Concretely, we develop mixture of disentangling deep generative models, where the mixture components correspond to object categories and each component model represents shape and view in a category-speciﬁc and mutually invariant manner.
Revised: Concretely, we develop mixture of disentangling deep generative models, where the mixture components correspond to object categories and each component model represents shape and view in a category-speciﬁc and mutually invariant manner.

Original: We devise a learning method based on variational autoencoders that does not explicitly use label information but uses only grouping information that links together different views of the same object.
Revised: We devise a learning method based on variational autoencoders that does not explicitly use label information but uses only grouping information that links together different views of the same object.

Original: Using several datasets of 3D objects including ShapeNet, we demonstrate that our model often outperforms previous relevant models including state-of-the-art methods in invariant clustering and one-shot classiﬁcation tasks, in a manner exposing the importance of categorical invariant representation.
Revised: Using several datasets of 3D objects including ShapeNet, we demonstrate that our model often outperforms previous relevant models including state-of-the-art methods in invariant clustering and one-shot classiﬁcation tasks, in a manner exposing the importance of categorical invariant representation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In everyday life, we see objects in a great variety.
Revised: In everyday life, we see objects in a great variety.

Original: Categories of objects are numerous and their shape variations are tremendously rich; different views make an object look totally different (Figure 1(A)).
Revised: Categories of objects are numerous and their shape variations are tremendously rich; different views make an object look totally different (Figure 1(A)).

Original: Recent neuroscientiﬁc studies have revealed how the primate brain organizes representation of complex objects in the higher visual cortex (Freiwald & Tsao, 2010;
Revised: Recent neuroscientiﬁc studies have revealed how the primate brain organizes representation of complex objects in the higher visual cortex (Freiwald & Tsao, 2010;

Original: Srihasam et al., 2014; Bao et al., 2020).
Revised: Srihasam et al., 2014; Bao et al., 2020).

Original: According to these, it comprises multi-stream networks, each of which is specialized to a particular object category, encodes category-speciﬁc visual features, and undergoes multiple stages with increasing view invariance.
Revised: According to these, it comprises multi-stream networks, each of which is specialized to a particular object category, encodes category-speciﬁc visual features, and undergoes multiple stages with increasing view invariance.

Original: These biological ﬁndings inspire us a new form of learning model that has multiple modules of category-speciﬁc invariant feature representations.
Revised: These biological ﬁndings inspire us a new form of learning model that has multiple modules of category-speciﬁc invariant feature representations.

Original: More speciﬁcally, our goal is, given an image dataset of general objects, to learn a generative model representing three latent factors: (1) category (e.g., cars, chairs), (2) shape (e.g., particular car or chair types), and (3) view (e.g., 3D orientation).
Revised: More speciﬁcally, our goal is, given an image dataset of general objects, to learn a generative model representing three latent factors: (1) category (e.g., cars, chairs), (2) shape (e.g., particular car or chair types), and (3) view (e.g., 3D orientation).

Original: A similar problem has been addressed by recent disentangling models that discover complex factors of input variations in a way invariant to each other (Tenenbaum & Freeman, 2000; Kingma et al., 2014;
Revised: A similar problem has been addressed by recent disentangling models that discover complex factors of input variations in a way invariant to each other (Tenenbaum & Freeman, 2000; Kingma et al., 2014;

Original: Chen et al., 2016; Higgins et al., 2016;
Revised: Chen et al., 2016; Higgins et al., 2016;

Original: Bouchacourt et al., 2018; Hosoya, 2019).
Revised: Bouchacourt et al., 2018; Hosoya, 2019).

Original: However, although these models can effectively infer a category or shape factor separately from a view factor, these typically cannot capture the structure in general object images that the diversity of shapes is much larger across categories than within a category.
Revised: However, although these models can effectively infer a category or shape factor separately from a view factor, these typically cannot capture the structure in general object images that the diversity of shapes is much larger across categories than within a category.

Original: In this study, we propose a novel model called CIGMO (Categorical Invariant Generative MOdel), which can learn to represent all the three factors (category, shape, and view) at once only with weak supervision.
Revised: In this study, we propose a novel model called CIGMO (Categorical Invariant Generative MOdel), which can learn to represent all the three factors (category, shape, and view) at once only with weak supervision.

Original: Our model has the form of mixture of deep generative models, where the mixture components correspond to categories and each component model gives a disentangled representation of shape and view for a particular category.
Revised: Our model has the form of mixture of deep generative models, where the mixture components correspond to categories and each component model gives a disentangled representation of shape and view for a particular category.

Original: We develop a learning algorithm based on variational autoencoders (VAE) method (Kingma & Welling, 2014) that does not use explicit labels, but uses only grouping information that links together different views of the same object (Bouchacourt et al., 2018;
Revised: We develop a learning algorithm based on variational autoencoders (VAE) method (Kingma & Welling, 2014) that does not use explicit labels, but uses only grouping information that links together different views of the same object (Bouchacourt et al., 2018;

Original: Chen et al., 2018; Hosoya, 2019).
Revised: Chen et al., 2018; Hosoya, 2019).

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: v a r i a t i
Revised: v a r i a t i

Original: [Figure]
Revised: [Figure]

Original: (B
Revised: (B

Original: [Figure]
Revised: [Figure]

Original: CIGMO (C) yyxxyy 2 ...
Revised: CIGMO (C) yyxxyy 2 ...

Original: ... xxy
Revised: ... xxy

Original: K y K x K x K
Revised: K y K x K x K

Original: [Figure]
Revised: [Figure]

Original: Using two image datasets of 3D objects (one derived from ShapeNet (Chang et al., 2015)), we demonstrate that CIGMO can solve multiple unconventional visual tasks on objects that are unseen during training.
Revised: Using two image datasets of 3D objects (one derived from ShapeNet (Chang et al., 2015)), we demonstrate that CIGMO can solve multiple unconventional visual tasks on objects that are unseen during training.

Original: These include invariant clustering, i.e., clustering of objects regardless of the view, one-shot classiﬁcation, i.e., object recognition given one example per class, and other various feature manipulations using the disentangled representation.
Revised: These include invariant clustering, i.e., clustering of objects regardless of the view, one-shot classiﬁcation, i.e., object recognition given one example per class, and other various feature manipulations using the disentangled representation.

Original: Quantitative comparison indicates that our model often outperforms many existing approaches including state-of-the-art methods.
Revised: Quantitative comparison indicates that our model often outperforms many existing approaches including state-of-the-art methods.

Original: Our key contributions are (1) development of the new deep generative model CIGMO, together with the VAE-based weakly supervised learning algorithm, (2) experiments of the model on two datasets with quantitative comparisons that show performance advantages over several existing models, and (3) empirical exposition of the importance of separate representation of category, shape, and view factors in modeling general object images.
Revised: Our key contributions are (1) development of the new deep generative model CIGMO, together with the VAE-based weakly supervised learning algorithm, (2) experiments of the model on two datasets with quantitative comparisons that show performance advantages over several existing models, and (3) empirical exposition of the importance of separate representation of category, shape, and view factors in modeling general object images.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The present work is closely related to recently proposed disentangling models for discovering mutually invariant factors of variation in the input.
Revised: The present work is closely related to recently proposed disentangling models for discovering mutually invariant factors of variation in the input.

Original: In one direction, some models have used unsupervised learning with certain constraints on the latent variable, though these seem to be effective only in limited cases (Higgins et al., 2016; Chen et al., 2016).
Revised: In one direction, some models have used unsupervised learning with certain constraints on the latent variable, though these seem to be effective only in limited cases (Higgins et al., 2016; Chen et al., 2016).

Original: Thus, more practical approaches have made use of explicit labels, such as semi-supervised using class labels for a part of dataset (Kingma et al., 2014; Siddharth et al., 2017) or adversarial learning to promote disentanglement (Lample et al., 2017; Mathieu et al., 2016).
Revised: Thus, more practical approaches have made use of explicit labels, such as semi-supervised learning for a part of dataset (Kingma et al., 2014; Siddharth et al., 2017) or adversarial learning to promote disentanglement (Lample et al., 2017; Mathieu et al., 2016).

Original: However, labels are often expensive.
Revised: However, labels are often expensive.

Original: To ﬁnd out a good compromise, weaker forms of supervision have been investigated.
Revised: To ﬁnd out a good compromise, weaker forms of supervision have been investigated.

Original: One such direction is group-based learning, which assumes inputs with the same shape to be grouped together (Bouchacourt et al., 2018; Chen et al., 2018; Hosoya, 2019).
Revised: One such direction is group-based learning, which assumes inputs with the same shape to be grouped together (Bouchacourt et al., 2018; Chen et al., 2018; Hosoya, 2019).

Original: In particular, our study here is technically much inﬂuenced by Group-based VAE (GVAE) (Hosoya, 2019) in the algorithm construction (Section 3).
Revised: In particular, our study here is technically much inﬂuenced by Group-based VAE (GVAE) (Hosoya, 2019) in the algorithm construction (Section 3).

Original: However, these  group-based methods are fundamentally limited in that the factors that can be separated are two—a group-common factor (shape) and an instance-speciﬁc factor (view)— and there is no obvious way to extend it to more than two.
Revised: However, these existing group-based methods are fundamentally limited in that the factors that can be separated are two—a group-common factor (shape) and an instance-speciﬁc factor (view)—and there is no obvious way to extend it to more than two.

Original: Thus, our novelty here is to introduce a mixture model comprising multiple GVAE models (each with shape and view variables) so that ﬁtting the mixture model to a grouped dataset can give rise to the third factor, categories, as mixture components.
Revised: Thus, our novelty here is to introduce a mixture model comprising multiple GVAE models (each with shape and view variables) so that ﬁtting the mixture model to a grouped dataset can give rise to the third factor, categories, as mixture components.

Revised: In Section 4, we examine the empirical merits of this technique in several tasks.

Original: Note that grouping information can most naturally be found in temporal data (like videos) since the object identity is often stable over time, cf.
Revised: Note that grouping information can most naturally be found in temporal data (like videos) since the object identity is often stable over time, cf.

Original: classical temporal coherence principle (F ¨ oldi ´ ak, 1991).
Revised: classical temporal coherence principle (F ¨ oldi ´ ak, 1991).

Original: Indeed, some weakly supervised disentangling approaches have explicitly used such temporal structure (Yang et al., 2015).
Revised: Indeed, some weakly supervised disentangling approaches have explicitly used such temporal structure (Yang et al., 2015).

Original: Some recent work has used deep nets for clustering of complex object images.
Revised: Some recent work has used deep nets for clustering of complex object images.

Original: The most typical approach is a simple combination of a deep generative model (e.g., VAE) and a conventional clustering method (e.g., Gaussian mixture), although such approach seems to be limited in capturing large object view variation (Jiang et al., 2017).
Revised: The most typical approach is a simple combination of a deep generative model (e.g., VAE) and a conventional clustering method (e.g., Gaussian mixture), although such approach seems to be limited in capturing large object view variation (Jiang et al., 2017).

Original: A latest approach proposes a feedforward approach that takes pairs of image data, similarly to ours, and maximizes the mutual information between the categorical posterior probability distributions for such paired image; this has showed remarkable clustering performance on natural images under various view variation (Ji et al., 2019).
Revised: A latest approach proposes a feedforward approach that takes pairs of image data, similarly to ours, and maximizes the mutual information between the categorical posterior probability distributions for such paired image; this has shown remarkable clustering performance on natural images under various view variation (Ji et al., 2019).

Original: In Section 4, we experimentally compare their method with ours.
Revised: In Section 4, we experimentally compare their method with ours.

Original: Note, however, that these methods are specialized to clustering and throw away all information other than the category.
Revised: Note, however, that these methods are specialized to clustering and throw away all information other than the category.

Original:  CIGMO: C ATEGORICAL I NVARIANT G ENERATIVE M ODEL 3. M ODEL
Revised:  CIGMO: C ATEGORICAL I NVARIANT G ENERATIVE M ODEL 3. M ODEL

Original: In our framework, we assume a grouped dataset
Revised: In our framework, we assume a grouped dataset

Original: [Equation]
Revised: [Equation]

Original: where each data point is a group (tuple) of K data instances (e.g., images); we assume independence between groups but not instances within a group.
Revised: where each data point is a group (tuple) of K data instances (e.g., images); we assume independence between groups but not instances within a group.

Original: For a data group ( x 1 , . . .
Revised: For a data group ( x 1 , . . .

Original: , x K ) , we consider three types of hidden variables: category c ∈ { 1 , . .
Revised: , x K ) , we consider three types of hidden variables: category c ∈ { 1 , . .

Original: , C } , shape z ∈ R M , and views y 1 , . . .
Revised: , C } , shape z ∈ R M , and views y 1 , . . .

Original: , y K ∈ R L (eliding the superscript ( n ) for brevity), where the category and shape are common for the group while the views are speciﬁc to each instance.
Revised: , y K ∈ R L (eliding the superscript ( n ) for brevity), where the category and shape are common for the group while the views are speciﬁc to each instance.

Original: We consider the following generative model (Figure 1(B)):
Revised: We consider the following generative model (Figure 1(B)):

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) for c = 1 , . . . , C and k = 1 , . .
Revised: for c = 1 , . . . , C and k = 1 , . .

Original: , K .
Revised: , K .

Original: Here, f c is a decoder deep net deﬁned for each category c and π c is a category prior with (cid:80) Cc
Revised: Here, f c is a decoder deep net deﬁned for each category c and π c is a category prior with (cid:80) Cc

Original: =1 π c = 1 .
Revised: =1 π c = 1 .

Original: In the generative process, ﬁrst, the category c is drawn from the categorical distribution ( π 1 , . . .
Revised: In the generative process, ﬁrst, the category c is drawn from the categorical distribution ( π 1 , . . .

Original: , π C ) , while the shape z and views y k are drawn from standard
Revised: , π C ) , while the shape z and views y k are drawn from standard

Original: Gaussian priors.
Revised: Gaussian priors.

Original: Then, each data instance x k is generated by the decoder deep net f c for category c applied to the group-common shape z and the instance-speciﬁc view y k (added with Gaussian noise of unit variance).
Revised: Then, each data instance x k is generated by the decoder deep net f c for category c applied to the group-common shape z and the instance-speciﬁc view y k (added with Gaussian noise of unit variance).

Original: In other words, the decoder f c generates different data instances in a group from the same shape and different views.
Revised: In other words, the decoder f c generates different data instances in a group from the same shape and different views.

Original: Having deﬁned a mixture of deep generative models as above, we expect that, after ﬁtting it to a view-grouped object image dataset, object categories will arise as mixture components and category-speciﬁc shapes and views will be represented in each component model.
Revised: Having deﬁned a mixture of deep generative models as above, we expect that, after ﬁtting it to a view-grouped object image dataset, object categories will arise as mixture components and category-speciﬁc shapes and views will be represented in each component model.

Original: 3. L EARNING
Revised: 3. L EARNING

Original: We construct a learning algorithm following the VAE approach (Kingma & Welling, 2014).
Revised: We construct a learning algorithm following the VAE approach (Kingma & Welling, 2014).

Original: As the most important step, we specify deep net architectures to encode approximate posterior distributions (Figure 1(C)).
Revised: As the most important step, we specify inference models to encode approximate posterior distributions (Figure 1(C)).

Original: First, we estimate the posterior probability for category c as follows:
Revised: First, we estimate the posterior probability for category c as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, u is a classiﬁer deep net that takes an individual instance x and outputs a probability distribution over the categories ( (cid:80) Cc
Revised: Here, u is a classiﬁer deep net that takes an individual instance x and outputs a probability distribution over the categories ( (cid:80) Cc

Original: =1 u ( c ) ( x ) = 1 ), similarly to (Kingma et al., 2014).
Revised: =1 u ( c ) ( x ) = 1 ), similarly to (Kingma et al., 2014).

Original: We then take the average over the instance-speciﬁc probability distributions and use it as the group-common distribution.
Revised: We then take the average over the instance-speciﬁc probability distributions and use it as the group-common distribution.

Original: This is a simple approach to make the instance-speciﬁc distributions converge to equal values, i.e., u ( x 1 )
Revised: This is a simple approach to make the instance-speciﬁc distributions converge to equal values, i.e., u ( x 1 )

Original: ≈ u
Revised: ≈ u

Original: ( x 2 ) , an adaptation of a key technique of GVAE used for computing the group-common shape representation (Hosoya, 2019), which is also used below.
Revised: ( x 2 ) . This is an adaptation of a key technique of GVAE used for computing the group-common shape representation (Hosoya, 2019); see below.

Original: Then, given the estimated category c , we infer each instance-speciﬁc view y k from the input x k as follows:
Revised: Then, given the estimated category c , we infer each instance-speciﬁc view y k from the input x k as follows:

Original: [Equation]
Revised: [Equation]

Original: where g c and r c are encoder deep nets that are deﬁned for each category c to specify the mean and variance, respectively.
Revised: where g c and r c are encoder deep nets that are deﬁned for each category c to specify the mean and variance, respectively.

Original: To estimate shape z , we compute the following:
Revised: To estimate shape z , we compute the following:

Original: [Equation]
Revised: [Equation]

Original: Here, again, encoder deep nets h c and s c are deﬁned for each category c .
Revised: Here, again, encoder deep nets h c and s c are deﬁned for each category c .

Original: These compute the mean and variance, respectively, of the posterior distribution for the individual shape for each instance x k .
Revised: These compute the mean and variance, respectively, of the posterior distribution for the individual shape for each instance x k .

Original: Then, the group-common shape z is obtained as the average over all the individual shapes.
Revised: Then, the group-common shape z is obtained as the average over all the individual shapes.

Original: In this way, again, the instance-speciﬁc shape representations are expected to converge to an equal value in the course of training, i.e., h c ( x 1 ) ≈ h c ( x 2 ) (Hosoya, 2019).
Revised: In this way, again, the instance-speciﬁc shape representations are expected to converge to an equal value in the course of training, i.e., h c ( x 1 ) ≈ h c ( x 2 ) (Hosoya, 2019).

Original: Note that the way the view and shape are inferred  is  borrowed from Hosoya (2019).
Revised: Note that the way the view and shape are inferred here is mostly borrowed from Hosoya (2019).

Original: For training, we deﬁne the following variational lower bound of the marginal log likelihood for a data point:
Revised: For training, we deﬁne the following variational lower bound of the marginal log likelihood for a data point:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with the set φ of all weight parameters in the classiﬁer, encoder, and decoder deep nets.
Revised: with the set φ of all weight parameters in the classiﬁer, encoder, and decoder deep nets.

Original: We compute the reconstruction term L recon as follows:L
Revised: We compute the reconstruction term L recon as follows:

Original: [Equation]
Revised: [Equation]

Original: where we approximate the expectation using one sample z ∼ q ( z | x 1 , .
Revised: where we approximate the expectation using one sample z ∼ q ( z | x 1 , .

Original: , x K , c ) and y k ∼ q ( y k | x k , c ) for each k , but directly use the probability value q ( c | x 1 , . . .
Revised: , x K , c ) and y k ∼ q ( y k | x k , c ) for each k , but directly use the probability value q ( c | x 1 , . . .

Original: , x K ) for c .
Revised: , x K ) for c .

Original: The KL term
Revised: The KL term

Original: L KL is computed as follows:
Revised: L KL is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: Finally, our training procedure is to maximize the lower bound for the entire dataset: ˆ φ = arg max φ 1 N (cid:80)Nn =1 L ( φ ; x ( n ) 1 , . . .
Revised: Finally, our training procedure is to maximize the lower bound for the entire dataset with respect to the weight parameters: ˆ φ = arg max φ 1 N (cid:80)

Original:  , x ( n ) K ) with respect to the deep net weights.
Revised: Nn =1 L ( φ ; x ( n ) 1 , . . . , x ( n ) K ) .

Original: A diagrammatic outline of the algorithm is given in Figure 2.
Revised: A diagrammatic outline of the algorithm is given in Figure 2.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We have applied the model described in Section 3 to two image datasets: ShapeNet (general objects) and MultiPie (natural faces).
Revised: We have applied the model described in Section 3 to two image datasets: ShapeNet (general objects) and MultiPie (natural faces).

Original: Below, we outline the experimental set-up and show the results.
Revised: Below, we outline the experimental set-up and show the results.

Original: 4. S HAPE N ET
Revised: 4. S HAPE N ET

Original: For the ﬁrst set of experiment, we created a dataset of multi-viewed object images derived from 3D models in ShapeNet database (Chang et al., 2015).
Revised: For the ﬁrst set of experiment, we created a dataset of multi-viewed object images derived from 3D models in ShapeNet database (Chang et al., 2015).

Original: We selected 10, out of 55, pre-deﬁned object classes that each have a relatively large number of object identities (car, chair, table, airplane, lamp,
Revised: We selected 10, out of 55, pre-deﬁned object classes that each have a relatively large number of object identities (car, chair, table, airplane, lamp,

Original: [Figure]
Revised: [Figure]

Original: shape
Revised: shape

Original: [Figure]
Revised: [Figure]

Original: probabilities
Revised: probabilities

Original: [Figure]
Revised: [Figure]

Original: data group xyyz z h 1 /sh C /s C u g 1 /rg
Revised: data group xyyz z h 1 /sh C /s C u g 1 /rg

Original: C /r C ff C ()average average average ...
Revised: C /r C ff C ()average average average ...

Original: C ()loss boat, box, display, truck, and vase); we avoided heavily overlapping classes with many visually similar objects, e.g., chair, sofa, and bench.
Revised: C ()loss boat, box, display, truck, and vase); we avoided heavily overlapping classes with many visually similar objects, e.g., chair, sofa, and bench.

Original: We then rendered each object in 30 views in a single lighting condition.
Revised: We then rendered each object in 30 views in a single lighting condition.

Original: We split the training and test sets, which consisted of 21888 and 6210 object identities, respectively.
Revised: We split the training and test sets, which consisted of 21888 and 6210 object identities, respectively.

Original: We also created subset versions with 2, 3, or 5 object classes.
Revised: We also created subset versions with 2, 3, or 5 object classes.

Original: For training data, we formed groups of images of the same object in random 3 views ( K = 3 ).
Revised: For training data, we formed groups of images of the same object in random 3 views ( K = 3 ).

Original: We used object identity labels (not class labels) for forming grouped data, but, after this step, we never used any label during the training.
Revised: We used object identity labels (not class labels) for forming grouped data, but, after this step, we never used any label during the training.

Original: We converted all images to gray-scale to make the model concentrate on shape information rather than color information, as the latter easily becomes obvious clues for clustering (Ji et al., 2019).
Revised: We converted all images to gray-scale to make the model concentrate on shape information rather than color information, as the latter easily becomes obvious clues for clustering (Ji et al., 2019).

Original: See Appendix A for more detail on the dataset.
Revised: See Appendix A for more detail on the dataset.

Original: To train a CIGMO model, we used the following architecture.
Revised: To train a CIGMO model, we used the following architecture.

Original: First, we set the number of categories in the model to the number of classes in the data ( C = 2 , 3 , 5 , or 10 ).
Revised: First, we set the number of categories in the model to the number of classes in the data ( C = 2 , 3 , 5 , or 10 ).

Original: We set the shape dimension
Revised: We set the shape dimension

Original: M = 100 and the view dimension L = 3 .
Revised: M = 100 and the view dimension L = 3 .

Original: Here, using a very low view dimension was crucial since otherwise the view variable y k would take over all the information in the input and the shape variable z would become degenerate (Hosoya, 2019).
Revised: Here, using a very low view dimension was crucial since otherwise the view variable y k would take over all the information in the input and the shape variable z would become degenerate (Hosoya, 2019).

Original: The classiﬁer deep net u consisted of three convolutional layers and two fully connected layers and ended with a Softmax layer.
Revised: The classiﬁer deep net u consisted of three convolutional layers and two fully connected layers and ended with a Softmax layer.

Original: The shape and view encoder deep nets had a similar architecture, except that the last layer was linear for mean encoding ( g c and h c ) and ended with Softplus for variance encoding ( r c and s c ).
Revised: The shape and view encoder deep nets had a similar architecture, except that the last layer was linear for mean encoding ( g c and h c ) and ended with Softplus for variance encoding ( r c and s c ).

Original: The decoder deep nets f c had an inverse architecture ending with Sigmoid.
Revised: The decoder deep nets f c had an inverse architecture ending with Sigmoid.

Original: Since the model had so many deep nets, a large part of the networks was shared to save the memory space.
Revised: Since the model had so many deep nets, a large part of the networks was shared to save the memory space.

Original: See Appendix B for details of the architecture.
Revised: See Appendix B for details of the architecture.

Original: For simplicity, we ﬁxed the category prior π c = 1 /C .
Revised: For simplicity, we ﬁxed the category prior π c = 1 /C .

Original: For optimization, we used
Revised: For optimization, we used

Original: Adam (Kingma & Ba, 2015) with mini-batches of size 100 .
Revised: Adam (Kingma & Ba, 2015) with mini-batches of size 100 .

Original: In addition to CIGMO models, we trained a number of related models for comparison.
Revised: In addition to CIGMO models, we trained a number of related models for comparison.

Original: To investigate the effect of decoupled representation of category and shape, we trained GVAE models (Hosoya, 2019), which can be obtained as a special case of CIGMO with a single category ( C = 1 ).
Revised: To investigate the effect of decoupled representation of category and shape, we trained GVAE models (Hosoya, 2019), which can be obtained as a special case of CIGMO with a single category ( C = 1 ).

Original: To examine the effect of disentangling of shape and view, we trained mixture of VAEs, again obtained as CIGMO with no grouping ( K = 1 ; the shape and view variables are integrated).
Revised: To examine the effect of disentangling of shape and view, we trained mixture of VAEs, again obtained as CIGMO with no grouping ( K = 1 ; the shape and view variables are integrated).

Original: We also trained plain VAEs for basic comparison ( C = K = 1 ).
Revised: We also trained plain VAEs for basic comparison ( C = K = 1 ).

Original: For a part of evaluation below, since GVAE
Revised: For a part of evaluation below, since GVAE

Original: [Figure]
Revised: [Figure]

Original: M i x .
Revised: M i x .

Original: o f
Revised: o f

Original: VAE s
Revised: VAE s

Original: G
Revised: G

Original: VAE + k - m ean s
Revised: VAE + k - m ean s

Original: II C and VAE themselves have no clustering capability, we performed k -means on the shape variable of
Revised: II C and VAE themselves have no clustering capability, we performed k -means on the shape variable of

Original: GVAE or the latent variable of VAE.
Revised: GVAE or the latent variable of VAE.

Original: In addition, we incorporated two completely different methods:
Revised: In addition, we incorporated two completely different methods:

Original: Multi-Level VAE (MLVAE) (Bouchacourt et al., 2018), another group-based generative model, and
Revised: Multi-Level VAE (MLVAE) (Bouchacourt et al., 2018), another group-based generative model, and

Original: Invariant Information Clustering (IIC) (Ji et al., 2019), a method specialized to invariant clustering.
Revised: Invariant Information Clustering (IIC) (Ji et al., 2019), a method specialized to invariant clustering.

Original: We tested two versions of IIC, with and without regularization using 5-times overclustering (Ji et al., 2019).
Revised: We tested two versions of IIC, with and without regularization using 5-times overclustering (Ji et al., 2019).

Original: For each method, we trained 10 model instances from the scratch.
Revised: For each method, we trained 10 model instances from the scratch.

Original: We evaluated the trained models using test data, which were ungrouped and contained objects of the same classes as training data but of different identities.
Revised: We evaluated the trained models using test data, which were ungrouped and contained objects of the same classes as training data but of different identities.

Original: The evaluation involved three tasks: (1) invariant clustering, (2) one-shot classiﬁcation, and (3) feature manipulation.
Revised: The evaluation involved three tasks: (1) invariant clustering, (2) one-shot classiﬁcation, and (3) feature manipulation.

Original: 4.1. I NVARIANT CLUSTERING
Revised: 4.1. I NVARIANT CLUSTERING

Original: In this task, we simply inferred the most probable category from a given image, ˆ c = arg max c q ( c | x ) .
Revised: In this task, we simply inferred the most probable category from a given image, ˆ c = arg max c q ( c | x ) .

Original: Figure 3(A) illustrates results from a CIGMO model with 3 categories, where each box shows random images belonging to each estimated category.
Revised: Figure 3(A) illustrates results from a CIGMO model with 3 categories, where each box shows random images belonging to each estimated category.

Original: This demonstrates a very precise clustering of objects achieved by the model, which is quite remarkable given the large view variation and no category label used during training.
Revised: This demonstrates a very precise clustering of objects achieved by the model, which is quite remarkable given the large view variation and no category label used during training.

Original: Figure 3(B–D) shows analogous examples for alternative models (mixture of VAEs, GVAE with k -means, and IIC).
Revised: Figure 3(B–D) shows analogous examples for alternative models (mixture of VAEs, GVAE with k -means, and IIC).

Original: The result shows a clear degrade of performance.
Revised: The result shows a clear degrade of performance.

Original: [Table]
Revised: [Table]

Original: For comparison of the methods, we quantiﬁed the performance of invariant clustering in terms of classiﬁcation accuracy.
Revised: For comparison of the methods, we quantiﬁed the performance of invariant clustering in terms of classiﬁcation accuracy.

Original: Here, we used the best category-to-class assignment computed by the Hungarian algorithm (Munkres, 1957).
Revised: Here, we used the best category-to-class assignment computed by the Hungarian algorithm (Munkres, 1957).

Original: Table 1 summarizes the results.
Revised: Table 1 summarizes the results.

Original: Generally, CIGMO outperformed the other methods in all cases with a large margin.
Revised: Generally, CIGMO outperformed the other methods in all cases with a large margin.

Original: More speciﬁcally, ﬁrst, CIGMO always performed better than mixture of VAEs, showing the importance of shape-view disentangling.
Revised: More speciﬁcally, ﬁrst, CIGMO always performed better than mixture of VAEs, showing the importance of shape-view disentangling.

Original: Second, CIGMO was also always better than GVAE (or MLVAE) plus k -means, also showing the importance of category-shape decoupling.
Revised: Second, CIGMO was also always better than GVAE (and MLVAE) plus k -means, also showing the importance of category-shape decoupling.

Original: In other words, if shapes of all categories were packed into a single latent variable, category information could not be clearly represented.
Revised: In other words, if shapes of all categories were packed into a single latent variable, category information could not be clearly represented.

Original: These two points, taken together, emphasize the categorical and invariant nature of our model.
Revised: These two points, taken together, emphasize the categorical and invariant nature of our model.

Original: Third, CIGMO gave performance superior to IIC, surpassing the state-of-the-art method for this task.
Revised: Third, CIGMO gave performance superior to IIC, surpassing the state-of-the-art method for this task.

Original: This was the case both with and without overclustering; in fact, we could not ﬁnd a consistent improvement by overclustering in IIC, contrary to the claim by Ji et al.
Revised: This was the case both with and without overclustering; in fact, we could not ﬁnd a consistent improvement by overclustering in IIC, contrary to the claim by Ji et al.

Original: 4.1. O NE - SHOT CLASSIFICATION
Revised: 4.1. O NE - SHOT CLASSIFICATION

Original: In this task, we split test data into gallery and probe, where gallery holds one image for each object and probe the rest, and then identify the object of each probe image.
Revised: In this task, we split test data into gallery and probe, where gallery holds one image for each object and probe the rest, and then identify the object of each probe image.

Original: Note that our purpose here is not to infer the class but the object identity, unlike invariant clustering.
Revised: Note that our purpose here is not to infer the class but the object identity, unlike invariant clustering.

Original: Note also that, since the test objects are disjoint from the training objects, both gallery and probe images contain only unseen objects for the model.
Revised: Note also that, since the test objects are disjoint from the training objects, both gallery and probe images contain only unseen objects for the model.

Original: We used this task here since its performance can serve as a criterion for evaluating disentangled representations (Hosoya, 2019).
Revised: We used this task here since its performance can serve as a criterion for evaluating disentangled representations (Hosoya, 2019).

Original: The rationale is that, if shape code is perfectly invariant in view, then all images of the same object should be mapped to an identical point in the shape space.
Revised: The rationale is that, if shape code is perfectly invariant in view, then all images of the same object should be mapped to an identical point in the shape space.

Original: Thus, we compared overall accuracy of one-shot classiﬁcation for CIGMO and other models.
Revised: Thus, we compared overall accuracy of one-shot classiﬁcation for CIGMO and other models.

Original: For this, we performed a nearest-neighbor method according to cosine distance in the shape space.
Revised: For this, we performed a nearest-neighbor method according to cosine distance in the shape space.

Original: Here, the shape space was deﬁned depending on the method.
Revised: Here, the shape space was deﬁned depending on the method.

Original: For GVAE (or MLVAE), the shape variable z
Revised: For GVAE (or MLVAE), the shape variable z

Original: = h ( x ) directly deﬁned the shape space.
Revised: = h ( x ) directly deﬁned the shape space.

Original: For CIGMO, since the shape representation depended on the category, we ﬁrst constructed a C × M matrix Z such that Z c, ∗ =
Revised: For CIGMO, since the shape representation depended on the category, we ﬁrst constructed a C × M matrix Z such that Z c, ∗ =

Original: h c ( x ) for c
Revised: h c ( x ) for c

Original: = ˆ c and
Revised: = ˆ c and

Original: Z c, ∗ = 0 otherwise, and then ﬂattened the matrix to a vector.
Revised: Z c, ∗ = 0 otherwise, and then ﬂattened the matrix to a vector.

Original: This gave us category-dependent shape vectors that could be directly compared and, in particular, those of different categories would give cosine distance 1 , the maximum value.
Revised: This gave us category-dependent shape vectors that could be directly compared and, in particular, those of different categories would give cosine distance 1 , the maximum value.

Original: For VAE or mixture of VAEs, we used a similar scheme except that we used the entire latent variable in place of shape variable.
Revised: For VAE or mixture of VAEs, we used a similar scheme except that we used the entire latent variable in place of shape variable.

Original: Table 2 summarizes the results.
Revised: Table 2 summarizes the results.

Original: Overall, CIGMO performed the best among the compared methods in all cases.
Revised: Overall, CIGMO performed the best among the compared methods in all cases.

Original: In particular, it outperformed, by far, mixture of VAEs, showing the success of shape information disentangled from view.
Revised: In particular, it outperformed, by far, mixture of VAEs, showing the success of shape information disentangled from view.

Original: Further, CIGMO also performed signiﬁcantly better than GVAE and MLVAE, which indicates that shapes can be represented more efﬁciently with category specialization than without.
Revised: Further, CIGMO also performed signiﬁcantly better than GVAE and MLVAE, which indicates that shapes can be represented more efﬁciently with category specialization than without.

Original: These results, again, reveal the advantage of the categorical and invariant representations in modeling general object images.
Revised: These results, again, reveal the advantage of the categorical and invariant representations in modeling general object images.

Original: (Note also that the scores were remarkably high even for up to 6210-way classiﬁcation by one shot.)
Revised: (Note also that the scores were remarkably high even for up to 6210-way classiﬁcation by one shot.)

Original: 4.1. F EATURE MANIPULATION
Revised: 4.1. F EATURE MANIPULATION

Original: CIGMO provides various ways of manipulating latent feature representations and generating new images.
Revised: CIGMO provides various ways of manipulating latent feature representations and generating new images.

Original: These include (1) swapping, to generate an image from the view of one image and the shape of another, (2) interpolation, to generate an image from the shape and view that linearly
Revised: These include (1) swapping, to generate an image from the view of one image and the shape of another, (2) interpolation, to generate an image from the shape and view that linearly

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: interpolates those of two images, and (3) random generation, to generate an image from shape and view randomly drawn from Gaussian distributions.
Revised: interpolates those of two images, and (3) random generation, to generate an image from shape and view randomly drawn from Gaussian distributions.

Original: Analogous manipulations have commonly been used in previous studies (Mathieu et al., 2016; Bouchacourt et al., 2018; Hosoya, 2019), but our cases are conditioned on a category.
Revised: Analogous manipulations have commonly been used in previous studies (Mathieu et al., 2016; Bouchacourt et al., 2018; Hosoya, 2019), but our cases are conditioned on a category.

Original: Appendix C gives examples of these feature manipulations, in which we can qualitatively conﬁrm that our model performed these tasks as intended, e.g., reasonably clear alignment of views in rows and shapes in columns in swapping.
Revised: Appendix C gives examples of these feature manipulations, in which we can qualitatively conﬁrm that our model performed these tasks as intended, e.g., reasonably clear alignment of views in rows and shapes in columns in swapping.

Original: Although generating sharp images was not the focus here, improvement in this direction could be done, e.g., by incorporating adversarial learning for regularization (Mathieu et al., 2016).
Revised: Although generating sharp images was not the focus here, improvement in this direction could be done, e.g., by incorporating adversarial learning for regularization (Mathieu et al., 2016).

Original: Relevant to these, we quantitatively evaluated the quality of shape-view disentanglement in each category.
Revised: Relevant to these, we quantitatively evaluated the quality of shape-view disentanglement in each category.

Original: Speciﬁcally, we measured how much information the shape or view variable contained on object identity and, for this purpose, we trained two-layer neural networks on either variable for classiﬁcation.
Revised: Speciﬁcally, we measured how much information the shape or view variable contained on object identity and, for this purpose, we trained two-layer neural networks on either variable for classiﬁcation (Mathieu et al., 2016; Bouchacourt et al., 2018).

Original: A better disentangled representation was expected to give a higher accuracy from the shape variable and a lower accuracy from the view variable.
Revised: A better disentangled representation was expected to give a higher accuracy from the shape variable and a lower accuracy from the view variable.

Original: We performed this for each category using the belonging test images and took the average accuracy weighted by the number of those images.
Revised: We performed this for each category using the belonging test images and took the average accuracy weighted by the number of those images.

Original: Table 3 summarizes the results.
Revised: Table 3 summarizes the results.

Original: Overall, CIGMO gives fairly comparable quality to the existing disentangling methods.
Revised: Overall, CIGMO gives fairly comparable quality to the existing disentangling methods.

Original: 4. M ULTI P IE
Revised: 4. M ULTI P IE

Original: For the second set of experiment, we used a dataset of multi-viewed face images derived from
Revised: For the second set of experiment, we used a dataset of multi-viewed face images derived from

Original: MultiPie dataset (Gross et al., 2010).
Revised: MultiPie dataset (Gross et al., 2010).

Original: We followed the same data preparation as Hosoya (2019), except that we included the images in all lighting conditions and converted them to gray-scale.
Revised: We followed the same data preparation as Hosoya (2019), except that we included the images in all lighting conditions and converted them to gray-scale.

Original: We split the training and test sets consisting of disjoint 795 and 126 identities, and grouped together the training images that have the same identity, hair/cloth style, and expression.
Revised: We split the training and test sets consisting of disjoint 795 and 126 identities, and grouped together the training images that have the same identity, hair/cloth style, and expression.

Original: For this dataset, there was no pre-deﬁned class unlike ShapeNet, so we arbitrarily set the number of categories toin CIGMO models.
Revised: For this dataset, there was no pre-deﬁned class unlike ShapeNet, so we arbitrarily set the number of categories toin CIGMO models.

Original: Otherwise, the training condition was the same as before.
Revised: Otherwise, the training condition was the same as before.

Original: Figure 4 illustrates the results from a CIGMO model on (A) invariant clustering and (B) categorywise swapping, where only 3 categories were shown as the other 2 categories were degenerate to which no input belonged.
Revised: Figure 4 illustrates the results from a CIGMO model on (A) invariant clustering and (B) categorywise swapping, where only 3 categories were shown as the other 2 categories were degenerate to which no input belonged.

Original: By inspection, the 3 effective categories seemed to represent faces with long hair, round faces with short hair, and oval faces with short hair, respectively, in a view-invariant manner (although this observation was not veriﬁed since the dataset lacked relevant labels).
Revised: By inspection, the 3 effective categories seemed to represent faces with long hair, round faces with short hair, and oval faces with short hair, respectively, in a view-invariant manner (although this observation was not veriﬁed since the dataset lacked relevant labels).

Original: Other
Revised: Other

Original: CIGMO model instances trained in the same way had 1 to 3 effective categories and seemed to have slightly different categorization strategies, but generally based on hair length, hair color, facial aspect ratio, beard, or skin color; we could not ﬁnd categorization by expression.
Revised: CIGMO model instances trained in the same way had 1 to 3 effective categories and seemed to have slightly different categorization strategies, but generally based on hair length, hair color, facial aspect ratio, beard, or skin color; we could not ﬁnd categorization by expression.

Original: We also measured
Revised: We also measured

Original: [Figure]
Revised: [Figure]

Original: one-shot accuracy and shape-view disentangling quality.
Revised: one-shot accuracy and shape-view disentangling quality.

Original: However, CIGMO overall gave a similar performance to GVAE or MLVAE, indicating a lesser importance of category-shape decoupling in this task for this dataset.
Revised: However, CIGMO overall gave a similar performance to GVAE or MLVAE, indicating a lesser importance of category-shape decoupling in this task for this dataset.

Original: This is understandable since, in a sense, all faces look alike and therefore, in what way faces are categorized, cross-category diversity would not be so large compared to within-category diversity.
Revised: This is understandable since, in a sense, all faces look alike and therefore, in what way faces are categorized, cross-category diversity would not be so large compared to within-category diversity.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have proposed CIGMO as a deep generative model that can discover category, shape, and view factors from general object images.
Revised: In this paper, we have proposed CIGMO as a deep generative model that can discover category, shape, and view factors from general object images.

Original: The model has the form of mixture of disentangling generative models and comes with a VAE-based algorithm that requires no explicit label but only view-grouping information.
Revised: The model has the form of mixture of disentangling generative models and comes with a VAE-based algorithm that requires no explicit label but only view-grouping information.

Original: By application to two image datasets, we showed that our model can learn to represent categories in the mixture components and category-speciﬁc disentangled information of shape and view in each component model.
Revised: By application to two image datasets, we showed that our model can learn to represent categories in the mixture components and category-speciﬁc disentangled information of shape and view in each component model.

Original: We demonstrated that our model can outperform existing methods including state-of-the-art methods in invariant clustering and oneshot classiﬁcation tasks, emphasizing the importance of categorical invariant representation.
Revised: We demonstrated that our model can outperform existing methods including state-of-the-art methods in invariant clustering and oneshot classiﬁcation tasks, emphasizing the importance of categorical invariant representation.

Original: Future investigation will include improvement on image generation quality, category degeneracy, and scalability, and application to more realistic datasets.
Revised: Future investigation will include improvement on image generation quality, category degeneracy, and scalability, and application to more realistic datasets.

Original: Lastly, CIGMO’s biological relationship to the primate inferotemporal cortex would be interesting to pursue, as our present work was originally inspired so.
Revised: Lastly, CIGMO’s biological relationship to the primate inferotemporal cortex would be interesting to pursue, as our present work was originally inspired so.

Original: A D ATASET DETAILS
Revised: A D ATASET DETAILS

Original: Our object image dataset stemmed from a core subset of ShapeNet database of 3D object models (Chang et al., 2015) used in SHREC2016 challenge 1 .
Revised: Our object image dataset stemmed from a core subset of ShapeNet database of 3D object models (Chang et al., 2015) used in SHREC2016 challenge 1 .

Original: The subset contained 55 object classes and did not include material data.
Revised: The subset contained 55 object classes and did not include material data.

Original: Out of the 55 classes, we selected 10 classes: car, chair, table, airplane, lamp, boat, box, display, truck and vase.
Revised: Out of the 55 classes, we selected 10 classes: car, chair, table, airplane, lamp, boat, box, display, truck and vase.

Original: Our criterion here was to select classes with a large number of object identities but avoid including visually similar classes, e.g., chair, sofa and bench.
Revised: Our criterion here was to select classes with a large number of object identities but avoid including visually similar classes, e.g., chair, sofa and bench.

Original: We rendered each object in 30 views consisting of 15 azimuths (equally dividing 360 ◦ ) and 2 elevations ( 0 ◦ and 22 .
Revised: We rendered each object in 30 views consisting of 15 azimuths (equally dividing 360 ◦ ) and 2 elevations ( 0 ◦ and 22 .

Original: 5 ◦ downward) in a single lighting condition; the images were gray-scale and had size 64 × 64 pixels.
Revised: 5 ◦ downward) in a single lighting condition; the images were gray-scale and had size 64 × 64 pixels.

Original: All the rendering used Blender software 2 .
Revised: All the rendering used Blender software 2 .

Original: We divided the data into training and test following the split given in the original database.
Revised: We divided the data into training and test following the split given in the original database.

Original: We also created versions of the dataset consisting of the ﬁrst 2, 3, or 5 object classes listed above.
Revised: We also created versions of the dataset consisting of the ﬁrst 2, 3, or 5 object classes listed above.

Original: B A RCHITECTURE DETAILS
Revised: B A RCHITECTURE DETAILS

Original: In a CIGMO model, the classiﬁer deep net u consisted of three convolutional layers each with 32 , 64 , and 128 ﬁlters (kernel 5 × 5 ; stride 2 ; padding 2), followed by two fully connected layers each with 500 intermediate units and C output units.
Revised: In a CIGMO model, the classiﬁer deep net u consisted of three convolutional layers each with 32 , 64 , and 128 ﬁlters (kernel 5 × 5 ; stride 2 ; padding 2), followed by two fully connected layers each with 500 intermediate units and C output units.

Original: These layers were each intervened with Batch
Revised: These layers were each intervened with Batch

Original: Normalization and ReLU nonlinearity, except that the last layer ended with Softmax.
Revised: Normalization and ReLU nonlinearity, except that the last layer ended with Softmax.

Original: The shape and view encoder deep nets had a similar architecture, except that the last layer was linear for encoding the mean ( g c and h c ) or ended with Softplus for encoding the variance ( r c and s c ).
Revised: The shape and view encoder deep nets had a similar architecture, except that the last layer was linear for encoding the mean ( g c and h c ) or ended with Softplus for encoding the variance ( r c and s c ).

Original: The decoder deep nets f c had two fully connected layers ( 103 input units and 500 intermediate units) followed by three transposed convolutional layers  with  64  ﬁlters (kernel 6 × 6 ; stride 2 ; padding 2 ).
Revised: The decoder deep nets f c had two fully connected layers ( 103 input units and 500 intermediate units) followed by three transposed convolutional layers each with 128 , 64 , and 32 ﬁlters (kernel 6 × 6 ; stride 2 ; padding 2 ).

Original: These layers were again intervened with Batch Normalization and ReLU nonlinearity, but the last layer was Sigmoid.
Revised: These layers were again intervened with Batch Normalization and ReLU nonlinearity, but the last layer was Sigmoid.

Original: To save the memory space, the shape encoders shared the ﬁrst four layers for all categories and for mean and variance.
Revised: To save the memory space, the shape encoders shared the ﬁrst four layers for all categories and for mean and variance.

Original: The view encoders shared the entire architecture for all categories, but with a separate last layer for mean or variance speciﬁcation.
Revised: The view encoders shared the entire architecture for all categories, but with a separate last layer for mean or variance speciﬁcation.

Original: The decoders shared all but the ﬁrst layer for all categories.
Revised: The decoders shared all but the ﬁrst layer for all categories.

Original: In the quantitative comparison, we obtained a mixture of VAEs, a GVAE model, and a VAE model as a special case of CIGMO model.
Revised: In the quantitative comparison, we obtained a mixture of VAEs, a GVAE model, and a VAE model as a special case of CIGMO model.

Original: Namely, a mixture of VAEs was a CIGMO with no grouping ( K = 1 ), a GVAE was a CIGMO with a single category ( C = 1 ), and a VAE was a CIGMO with both constraints ( K = 1 and C = 1 ).
Revised: Namely, a mixture of VAEs was a CIGMO with no grouping ( K = 1 ), a GVAE was a CIGMO with a single category ( C = 1 ), and a VAE was a CIGMO with both constraints ( K = 1 and C = 1 ).

Original: In the case of no grouping, since no structure could differentiate the shape and view dimensions, we combined these into a single latent variable ofdimensions.
Revised: In the case of no grouping, since no structure could differentiate the shape and view dimensions, we combined these into a single latent variable ofdimensions.

Original: C A DDITIONAL RESULTS
Revised: C A DDITIONAL RESULTS

Original: We performed the following feature manipulation tasks introduced in Section 4.1.3.
Revised: We performed the following feature manipulation tasks introduced in Section 4.1.3.

Original: Swapping For a category c and for images x 1 and x 2 belonging to c , obtain x swap =
Revised: Swapping For a category c and for images x 1 and x 2 belonging to c , obtain x swap =

Original: f c ( y 1 , z 2 ) where y 

Revised: f c ( y 1 , z 2 ) where y

Original: =
Revised: =

Original: g c ( x 1 ) and z 2 =
Revised: g c ( x 1 ) and z 2 =

Original: h c ( x 2 ) .
Revised: h c ( x 2 ) .

Original: Interpolation For a category c and for images x 1 and x 2 belonging to c , obtain x interp =
Revised: Interpolation For a category c and for images x 1 and x 2 belonging to c , obtain x interp =

Original: f c ( α y 1 + (1 − α )
Revised: f c ( α y 1 + (1 − α )

Original: y 2 ,
Revised: y 2 ,

Original: β z 1 + (1 − β )
Revised: β z 1 + (1 − β )

Original: z 2 ) where y i
Revised: z 2 ) where y i

Original: =
Revised: =

Original: g c ( x i ) and z i =
Revised: g c ( x i ) and z i =

Original: h c ( x i ) with 0 ≤ α, β ≤ 1 and i = 1 , 2 .
Revised: h c ( x i ) with 0 ≤ α, β ≤ 1 and i = 1 , 2 .

Original: Random generation For a category c , obtain x rand =
Revised: Random generation For a category c , obtain x rand =

Original: f c ( y , z ) with y ∼ N L (0 , I ) and z ∼
Revised: f c ( y , z ) with y ∼ N L (0 , I ) and z ∼

Original: N M (0 , I ) .
Revised: N M (0 , I ) .

Original: Figure 5 shows examples of these feature manipulations on a 3-category CIGMO model trained on
Revised: Figure 5 shows examples of these feature manipulations on a 3-category CIGMO model trained on

Original: ShapeNet.
Revised: ShapeNet.

Original: As we can see, swapping gives a clear alignment of views in rows and shapes in columns.
Revised: As we can see, swapping gives a clear alignment of views in rows and shapes in columns.

Original: Interpolation gives smooth changes of images from one image to another in both shape and view dimensions.
Revised: Interpolation gives smooth changes of images from one image to another in both shape and view dimensions.

Original: Random generation gives new images most of which are recognizable as each category.
Revised: Random generation gives new images most of which are recognizable as each category.

Original: [Figure]
Revised: [Figure]