Original: C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING VIA G ENERATIVE A DVERSARIAL N ETWORKS
Revised: C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING VIA G ENERATIVE A DVERSARIAL N ETWORKS

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised ﬁne-grained class clustering is a practical yet challenging task due to the difﬁculty of feature representations learning of subtle object details.
Revised: Unsupervised ﬁne-grained class clustering is a practical yet challenging task due to the difﬁculty of feature representations learning of subtle object details.

Original: We introduce C3-GAN, a method that leverages the categorical inference power of
Revised: We introduce C3-GAN, a method that leverages the categorical inference power of

Original: InfoGAN with contrastive learning.
Revised: InfoGAN with contrastive learning.

Original: We aim to learn feature representations that encourage a dataset to form distinct cluster boundaries in the embedding space, while also maximizing the mutual information between the latent code and its image observation.
Revised: We aim to learn feature representations that encourage a dataset to form distinct cluster boundaries in the embedding space, while also maximizing the mutual information between the latent code and its image observation.

Original: Our approach is to train a discriminator, which is also used for inferring clusters, to optimize the contrastive loss, where image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs.
Revised: Our approach is to train a discriminator, which is also used for inferring clusters, to optimize the contrastive loss, where image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs.

Original: Speciﬁcally, we map the input of a generator, which was sampled from the categorical distribution, to the embedding space of the discriminator and let them act as a cluster centroid.
Revised: Speciﬁcally, we map the input of a generator, which was sampled from the categorical distribution, to the embedding space of the discriminator and let them act as a cluster centroid.

Original: In this way, C3-GAN succeeded in learning a clustering-friendly embedding space where each cluster is distinctively separable.
Revised: In this way, C3-GAN succeeded in learning a clustering-friendly embedding space where each cluster is distinctively separable.

Original: Experimental results show that C3-GAN achieved  state-of-the-art clustering performances on four ﬁne-grained image datasets, while also alleviating the mode collapse phenomenon.
Revised: Experimental results show that C3-GAN achieved the state-of-the-art clustering performance on four ﬁne-grained image datasets, while also alleviating the mode collapse phenomenon.

Original: Code is available at https://github.com/naver-ai/c3-gan .
Revised: Code is available at https://github.com/naver-ai/c3-gan .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unsupervised ﬁne-grained class clustering is a task that classiﬁes images of very similar objects (Benny & Wolf, 2020).
Revised: Unsupervised ﬁne-grained class clustering is a task that classiﬁes images of very similar objects (Benny & Wolf, 2020).

Original: While a line of works based on multiview-based self-supervised learning (SSL) methods (He et al., 2020; Chen et al., 2020; Grill et al., 2020) shows promising results on a conventional coarse-grained class clustering task (Van Gansbeke et al., 2020), it is difﬁcult for a ﬁne-grained class clustering task to beneﬁt from these methods for two reasons.
Revised: While a line of works based on multiview-based self-supervised learning (SSL) methods (He et al., 2020; Chen et al., 2020; Grill et al., 2020) shows promising results on a conventional coarse-grained class clustering task (Van Gansbeke et al., 2020), it is difﬁcult for a ﬁne-grained class clustering task to beneﬁt from these methods for two reasons.

Original: First, though it is more challenging and ambiguous for ﬁnding distinctions between ﬁne-grained classes, datasets for this type of task are difﬁcult to be large-scale.
Revised: First, though it is more challenging and ambiguous for ﬁnding distinctions between ﬁne-grained classes, datasets for this type of task are difﬁcult to be large-scale.

Original: We visualized this idea in Figure 1 which shows that ﬁnding distinctions between ﬁne-grained classes is more difﬁcult than doing so for coarse-grained object classes.
Revised: We visualized this idea in Figure 1 which shows that ﬁnding distinctions between ﬁne-grained classes is more difﬁcult than doing so for coarse-grained object classes.

Original: Second, the augmentation processes in these methods consider subtle changes in color or shape, that actually play an important role in differentiating between classes, as noisy factors.
Revised: Second, the augmentation processes in these methods consider subtle changes in color or shape, that actually play an important role in differentiating between classes, as noisy factors.

Original: Thus, it is required to ﬁnd an another approach for a ﬁne-grained class clustering task.
Revised: Thus, it is required to ﬁnd an another approach for a ﬁne-grained class clustering task.

Original: Generative adversarial networks (GAN) (Goodfellow et al., 2014) can be a solution as it needs to learn ﬁne details to generate realistic images.
Revised: Generative adversarial networks (GAN) (Goodfellow et al., 2014) can be a solution as it needs to learn ﬁne details to generate realistic images.

Original: A possible starting point could be InfoGAN (Chen et al., 2016), that succeeded in unsupervised categorical inference on MNIST dataset (LeCun et al., 2010) by maximizing the mutual information between the latent code and its observation.
Revised: A possible starting point could be InfoGAN (Chen et al., 2016), that succeeded in unsupervised categorical inference on MNIST dataset (LeCun et al., 2010) by maximizing the mutual information between the latent code and its observation.

Original: The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes.
Revised: The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes.

Original: FineGAN (Singh et al., 2019) extends InfoGAN by integrating the scene decomposition method into the framework, and learns three latent codes for the hierarchical scene generation.
Revised: FineGAN (Singh et al., 2019) extends InfoGAN by integrating the scene decomposition method into the framework, and learns three latent codes for the hierarchical scene generation.

Original: Each code, that is sampled from the independent uniform categorical distribution, is sequentially injected to multiple generators for a background, a super-class object, and a sub-class object image syntheses.
Revised: Each code, that is sampled from the independent uniform categorical distribution, is sequentially injected to multiple generators for a background, a super-class object, and a sub-class object image syntheses.

Original: FineGAN demonstrated that two latent codes for object image generations could be also utilized for clustering real images into their ﬁne-grained classes, outperforming conventional coarse-grained class clustering methods.
Revised: FineGAN demonstrated that two latent codes for object image generations could be also utilized for clustering real images into their ﬁne-grained classes, outperforming conventional coarse-grained class clustering methods.

Original: This result implies that extracting only foreground features is very helpful for the given task.
Revised: This result implies that extracting only foreground features is very helpful for the given task.

Original: However, FineGAN requires object bounding box annotations and additional training of classiﬁers, which greatly hinder its applicability.
Revised: However, FineGAN requires object bounding box annotations and additional training of classiﬁers, which greatly hinder its applicability.

Original: Also, the method lacks the ability of learning the distribution of real images, due to 
Revised: Also, the method lacks the ability of learning the distribution of real images, due to the mode collapse phenomenon (Higgins et al., 2017).

Original: [Figure]
Revised: [Figure]

Original: In this paper, we propose C onstrastive ﬁne-grained C lass C lustering GAN (C3-GAN) that deals with the aforementioned problems.
Revised: In this paper, we propose C onstrastive ﬁne-grained C lass C lustering GAN (C3-GAN) that deals with the aforementioned problems.

Original: We ﬁrst remove the reliance on human-annotated labels by adopting the method for unsupervised scene decomposition learning that is inspired by PerturbGAN (Bielski & Favaro, 2019).
Revised: We ﬁrst remove the reliance on human-annotated labels by adopting the method for unsupervised scene decomposition learning that is inspired by PerturbGAN (Bielski & Favaro, 2019).

Original: We then adopt contrastive learning method (van den Oord et al., 2018) for training a discriminator by deﬁning pairs of image-latent features that maximize the mutual information of the two as positive pairs and the rest as negative pairs.
Revised: We then adopt contrastive learning method (van den Oord et al., 2018) for training a discriminator by deﬁning pairs of image-latent features that maximize the mutual information of the two as positive pairs and the rest as negative pairs.

Original: This is based on the intuition that optimal results would be obtained if cluster centroids are distributed in a way that each cluster can be linearly separable (Wang & Isola, 2020).
Revised: This is based on the intuition that optimal results would be obtained if cluster centroids are distributed in a way that each cluster can be linearly separable (Wang & Isola, 2020).

Original: In speciﬁc, we map the input of a generator that was sampled from the categorical distribution onto the embedding space of the discriminator, and let it act as a cluster centroid that pulls features of images that were generated with its speciﬁc value.
Revised: In speciﬁc, we map the input of a generator that was sampled from the categorical distribution onto the embedding space of the discriminator, and let it act as a cluster centroid that pulls features of images that were generated with its speciﬁc value.

Original: Since the relations with other latent values are set as negative pairs, each cluster would be placed farther away from each other by the nature of the contrastive loss.
Revised: Since the relations with other latent values are set as negative pairs, each cluster would be placed farther away from each other by the nature of the contrastive loss.

Original: We have conducted experiments on four ﬁne-grained image datasets including CUB, Stanford Cars, Stanford Dogs and Oxford Flower, and demonstrated the effectiveness of C3-GAN by showing that it achieves the state-of-the-art clustering performances on all datasets.
Revised: We have conducted experiments on four ﬁne-grained image datasets including CUB, Stanford Cars, Stanford Dogs and Oxford Flower, and demonstrated the effectiveness of C3-GAN by showing that it achieves the state-of-the-art clustering performances on all datasets.

Original: Moreover, the induced embedding space of the discriminator turned out to be also helpful for alleviating the mode collapse issue.
Revised: Moreover, the induced embedding space of the discriminator turned out to be also helpful for alleviating the mode collapse issue.

Original: We conjecture that this is because the generator is additionally required to be able to synthesize a certain number of object classes with distinct characteristics.
Revised: We conjecture that this is because the generator is additionally required to be able to synthesize a certain number of object classes with distinct characteristics.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We propose a novel form of the information-theoretic regularization to learn a clusteringfriendly embedding space that leads a dataset to form distinct cluster boundaries without falling into a degenerated solution.
Revised: • We propose a novel form of the information-theoretic regularization to learn a clusteringfriendly embedding space that leads a dataset to form distinct cluster boundaries without falling into a degenerated solution.

Original: With this method, our C3-GAN achieved the state-ofthe-art ﬁne-grained class clustering performances on four ﬁne-grained image datasets.
Revised: With this method, our C3-GAN achieved the state-ofthe-art ﬁne-grained class clustering performances on four ﬁne-grained image datasets.

Original: • By adopting scene decomposition learning method that does not require any humanannotated labels, our method can be applied to more diverse datasets.
Revised: • By adopting scene decomposition learning method that does not require any humanannotated labels, our method can be applied to more diverse datasets.

Original: • Our method of training a discriminator is not only suitable for class clustering task, but also good at alleviating the mode collapse issue of GANs.
Revised: • Our method of training a discriminator is not only suitable for class clustering task, but also good at alleviating the mode collapse issue of GANs.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Unsupervised Clustering.
Revised: Unsupervised Clustering.

Original: Unsupervised clustering methods can be mainly categorized into the information theory-based and Expectation–Maximization(EM)-based approaches.
Revised: Unsupervised clustering methods can be mainly categorized into the information theory-based and Expectation–Maximization(EM)-based approaches.

Original: The ﬁrst approach aims to maximize the mutual information between original images and their augmentations to train a model in an end-to-end fashion (Ji et al., 2019; Zhong et al., 2020).
Revised: The ﬁrst approach aims to maximize the mutual information between original images and their augmentations to train a model in an end-to-end fashion (Ji et al., 2019; Zhong et al., 2020).

Original: To enhance the performance,
Revised: To enhance the performance,

Original: IIC (Ji et al., 2019) additionally trains the auxiliary branch using an unlabeled large dataset, and
Revised: IIC (Ji et al., 2019) additionally trains the auxiliary branch using an unlabeled large dataset, and

Original: DRC (Zhong et al., 2020) optimizes the contrastive loss on logit features for reducing the intra-class feature variation.
Revised: DRC (Zhong et al., 2020) optimizes the contrastive loss on logit features for reducing the intra-class feature variation.

Original: EM-based methods decouple the cluster assignment process (E-step) and the feature representations learning process (M-step) to learn more robust representations.
Revised: EM-based methods decouple the cluster assignment process (E-step) and the feature representations learning process (M-step) to learn more robust representations.

Original: Speciﬁcally, the feature representations are learnt by not only ﬁtting the proto-clustering results estimated in the expectation step, but also optimizing particular pretext tasks.
Revised: Speciﬁcally, the feature representations are learnt by not only ﬁtting the proto-clustering results estimated in the expectation step, but also optimizing particular pretext tasks.

Original: The clusters are either inferred by k -means clustering (Xie et al., 2016; Caron et al., 2018; Liu et al., 2020; Li et al., 2021) or training of an additional encoder (Dizaji et al., 2017).
Revised: The clusters are either inferred by k -means clustering (Xie et al., 2016; Caron et al., 2018; Liu et al., 2020; Li et al., 2021) or training of an additional encoder (Dizaji et al., 2017).

Original: However, many of these methods suffer from an uneven allocation issue of k -means clustering, which could result in a degenerate solution.
Revised: However, many of these methods suffer from an uneven allocation issue of k -means clustering, which could result in a degenerate solution.

Original: For this reason,
Revised: For this reason,

Original: SCAN (Van Gansbeke et al., 2020) proposes the novel objective function that does not require the k -means clustering process.
Revised: SCAN (Van Gansbeke et al., 2020) proposes the novel objective function that does not require the k -means clustering process.

Original: Based on the observation that K nearest neighbors of each feature point belong to the same ground-truth cluster with a high probability, it trains a classiﬁer that assigns identical cluster id to all nearest neighbors, which signiﬁcantly improved the clustering performance.
Revised: Based on the observation that K nearest neighbors of each feature point belong to the same ground-truth cluster with a high probability, it trains a classiﬁer that assigns identical cluster id to all nearest neighbors, which signiﬁcantly improved the clustering performance.

Original: Unlike these prior methods where the cluster centroids are not learnt in an end-to-end manner or not the subject of interest, C3-GAN investigate and try to improve their distribution for achieving better clustering performance.
Revised: Unlike these prior methods where the cluster centroids are not learnt in an end-to-end manner or not the subject of interest, C3-GAN investigate and try to improve their distribution for achieving better clustering performance.

Original: Unsupervised Scene Decomposition.
Revised: Unsupervised Scene Decomposition.

Original: Heavy cost of annotating segmentation masks have drawn active research efforts on developing unsupervised segmentation methods.
Revised: Heavy cost of annotating segmentation masks have drawn active research efforts on developing unsupervised segmentation methods.

Original: Some works address the task in the information-theoretic perspective (Ji et al., 2019; Savarese et al., 2021), but a majority of the works are based on GANs.
Revised: Some works address the task in the information-theoretic perspective (Ji et al., 2019; Savarese et al., 2021), but a majority of the works are based on GANs.

Original: ReDO (Chen et al.
Revised: ReDO (Chen et al.

Original: (2019)) learns to infer object masks by redrawing an input scene, and PerturbGAN (Bielski & Favaro (2019)), that has a separate background and foreground generator, triggers scene decomposition by randomly perturbing foreground images.
Revised: (2019)) learns to infer object masks by redrawing an input scene, and PerturbGAN (Bielski & Favaro (2019)), that has a separate background and foreground generator, triggers scene decomposition by randomly perturbing foreground images.

Original: Meanwhile, another line of works utilizes pre-trained high quality generative models such as StyleGAN (Karras et al., 2019; 2020) and
Revised: Meanwhile, another line of works utilizes pre-trained high quality generative models such as StyleGAN (Karras et al., 2019; 2020) and

Original: BigGAN (Brock et al., 2019).
Revised: BigGAN (Brock et al., 2019).

Original: Labels4Free (Abdal et al., 2021) trains a segmentation network on top of the pre-trained StyleGAN, utilizing the fact that inputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis.
Revised: Labels4Free (Abdal et al., 2021) trains a segmentation network on top of the pre-trained StyleGAN, utilizing the fact that inputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis.

Original: Voynov &
Revised: Voynov &

Original: Babenko (2020) and Melas-Kyriazi et al.
Revised: Babenko (2020) and Melas-Kyriazi et al.

Original: (2021) explore the latent space of pre-trained GANs to ﬁnd the perturbing directions that can be used for inducing foreground masks.
Revised: (2021) explore the latent space of pre-trained GANs to ﬁnd the perturbing directions that can be used for inducing foreground masks.

Original: While all these methods aim to infer foreground masks by training an additional mask predictor with synthesized data, or projecting real images into the latent space of the pre-trained GAN, C3-GAN is rather focusing on the semantic representations learning of foreground objects.
Revised: While all these methods aim to infer foreground masks by training an additional mask predictor with synthesized data, or projecting real images into the latent space of the pre-trained GAN, C3-GAN is rather focusing on the semantic representations learning of foreground objects.

Original: Fine-grained Feature Learning.
Revised: Fine-grained Feature Learning.

Original: For ﬁne-grained feature representations learning, some works (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020) have extended InfoGAN by integrating the scene decomposition learning method into the framework with multiple pairs of adversarial networks.
Revised: For ﬁne-grained feature representations learning, some works (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020) have extended InfoGAN by integrating the scene decomposition learning method into the framework with multiple pairs of adversarial networks.

Original: FineGAN (Singh et al., 2019) learns multiple latent codes to use them for sequential generation of a background, a super-class object, and a sub-class object image, respectively.
Revised: FineGAN (Singh et al., 2019) learns multiple latent codes to use them for sequential generation of a background, a super-class object, and a sub-class object image, respectively.

Original: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020) extend FineGAN with multiple encoders to directly infer these latent codes from real images and use them for manipulating images.
Revised: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020) extend FineGAN with multiple encoders to directly infer these latent codes from real images and use them for manipulating images.

Original: Since this autoencoder-based structure could evoke a degenerate solution where only one generator is responsible for synthesizing the entire image, MixNMatch conducts adversarial learning on the joint distribution of the latent code and image (Donahue et al., 2017), while OneGAN trains a model in two stages by training generators ﬁrst and encoders next.
Revised: Since this autoencoder-based structure could evoke a degenerate solution where only one generator is responsible for synthesizing the entire image, MixNMatch conducts adversarial learning on the joint distribution of the latent code and image (Donahue et al., 2017), while OneGAN trains a model in two stages by training generators ﬁrst and encoders next.

Original: Even though these works have succeeded in generating images in a hierarchical manner and learnt latent codes that can be used for clustering real images corresponding to their ﬁne-grained classes, they cannot be applied to datasets that have no object bounding box annotations.
Revised: Even though these works have succeeded in generating images in a hierarchical manner and learnt latent codes that can be used for clustering real images corresponding to their ﬁne-grained classes, they cannot be applied to datasets that have no object bounding box annotations.

Original: Moreover, they require a training of additional classiﬁers using a set of generated images annotated with their latent values.
Revised: Moreover, they require a training of additional classiﬁers using a set of generated images annotated with their latent values.

Original: The proposed model in this paper, C3-GAN, can learn the latent code of foreground region in a completely unsupervised way, and simply utilize the discriminator for inferring the clusters of a given dataset.
Revised: The proposed model in this paper, C3-GAN, can learn the latent code of foreground region in a completely unsupervised way, and simply utilize the discriminator for inferring the clusters of a given dataset.

Original:  M ETHOD
Revised:  M ETHOD

Original: Given a dataset X = { x i } N − 1 i =

Revised: Given a dataset X = { x i } N − 1 i =

Original: consisting of single object images, we aim to distribute data into Y semantically ﬁne-grained classes.
Revised: consisting of single object images, we aim to distribute data into Y semantically ﬁne-grained classes.

Original: Our GAN-based model infers the clusters of data in the semantic feature space H ∈ R d h of the discriminator D .
Revised: Our GAN-based model infers the clusters of data in the semantic feature space H ∈ R d h of the discriminator D .

Original: The feature space H is learnt by maximizing the mutual information between the latent code, which is the input of a generator, and its image observation ˆ x .
Revised: The feature space H is learnt by maximizing the mutual information between the latent code, which is the input of a generator, and its image observation ˆ x .

Original: For more robust feature representations learning, we decompose a scene into a background and foreground region and associate the latent code mainly with the foreground region.
Revised: For more robust feature representations learning, we decompose a scene into a background and foreground region and associate the latent code mainly with the foreground region.

Original: We especially reformulate the information-theoretic regularization to optimize the contrastive loss deﬁned for latent-image feature pairs to induce each cluster to be linearly separated in the feature space.
Revised: We especially reformulate the information-theoretic regularization to optimize the contrastive loss deﬁned for latent-image feature pairs to induce each cluster to be linearly separated in the feature space.

Original: 𝒄 𝒛 𝑮 𝒇𝒈 𝑮 𝒃𝒈 !
Revised: 𝒄 𝒛 𝑮 𝒇𝒈 𝑮 𝒃𝒈 !

Original: 𝑥 !
Revised: 𝑥 !

Original: 𝑥 !" !
Revised: 𝑥 !" !

Original: 𝑥 #" ⨂ 𝒯 $ RandomMatrix !
Revised: 𝑥 #" ⨂ 𝒯 $ RandomMatrix !

Original: 𝑥 #" ′ ⨁ 𝑫 𝒃𝒂𝒔𝒆 𝑵(𝟎,𝑰) 𝓤(𝟎,𝒀 - 𝟏)
Revised: 𝑥 #" ′ ⨁ 𝑫 𝒃𝒂𝒔𝒆 𝑵(𝟎,𝑰) 𝓤(𝟎,𝒀 - 𝟏)

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER 1 CLUSTER CENTROID FEATURE ( 𝒍 ) IMAGE FEATURE ( 𝒉 )
Revised: CLUSTER 1 CLUSTER CENTROID FEATURE ( 𝒍 ) IMAGE FEATURE ( 𝒉 )

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER 1 𝝍 𝒄 𝝍 𝒙𝒉 𝝍 𝒙𝒓 𝝍 𝒙𝒉 𝑫 𝒃𝒂𝒔𝒆
Revised: CLUSTER 1 𝝍 𝒄 𝝍 𝒙𝒉 𝝍 𝒙𝒓 𝝍 𝒙𝒉 𝑫 𝒃𝒂𝒔𝒆

Original: REALorFAKE?
Revised: REALorFAKE?

Original: ( 𝒓 )
Revised: ( 𝒓 )

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: Our proposed method is built on FineGAN (Singh et al., 2019), which is in turn based on InfoGAN (Chen et al., 2016).
Revised: Our proposed method is built on FineGAN (Singh et al., 2019), which is in turn based on InfoGAN (Chen et al., 2016).

Original: InfoGAN learns to associate the latent code c with its observation ˆ x by inducing the model to maximize the mutual information of the two, I( c, ˆ x ) .
Revised: InfoGAN learns to associate the latent code c with its observation ˆ x by inducing the model to maximize the mutual information of the two, I( c, ˆ x ) .

Original: The latent code c can take various form depending on the prior knowledge of the factor that we want to infer, and is set to follow the uniform categorical distribution when our purpose is to make the categorical inference on given dataset.
Revised: The latent code c can take various form depending on the prior knowledge of the factor that we want to infer, and is set to follow the uniform categorical distribution when our purpose is to make the categorical inference on given dataset.

Original: FineGAN learns three such latent codes for hierarchical image generation, and each code is respectively used for background, super-class object, and sub-class object image synthesis.
Revised: FineGAN learns three such latent codes for hierarchical image generation, and each code is respectively used for background, super-class object, and sub-class object image synthesis.

Original: To facilitate this decomposition, FineGAN employs multiple pairs of generator and discriminator and trains an auxiliary background classiﬁer using object bounding box annotations.
Revised: To facilitate this decomposition, FineGAN employs multiple pairs of generator and discriminator and trains an auxiliary background classiﬁer using object bounding box annotations.

Original: It further demonstrates that the latent codes for object image synthesis can be also utilized for clustering an image dataset according to their ﬁne-grained classes.
Revised: It further demonstrates that the latent codes for object image synthesis can be also utilized for clustering an image dataset according to their ﬁne-grained classes.

Original: Our method differs from FineGAN in that, i) it employs the discriminator D to infer clusters without requiring additional training of classiﬁers, and ii) it learns only one latent code c that corresponds to the ﬁne-grained class of a foreground object.
Revised: Our method differs from FineGAN in that, i) it employs the discriminator D to infer clusters without requiring additional training of classiﬁers, and ii) it learns only one latent code c that corresponds to the ﬁne-grained class of a foreground object.

Original: The separate random noise z is kept, that is another input of the generator, to model variations occurring in a background region.
Revised: The separate random noise z is kept, that is another input of the generator, to model variations occurring in a background region.

Original: The noise value
Revised: The noise value

Original: z ∈ R d z is sampled from the normal distribution N (0 , I ) , and the latent code c ∈ R Y is an 1-hot vector where the index k that makes c k = 1 is sampled from the uniform categorical distribution, U (0 , Y − 1) .
Revised: z ∈ R d z is sampled from the normal distribution N (0 , I ) , and the latent code c ∈ R Y is an 1-hot vector where the index k that makes c k = 1 is sampled from the uniform categorical distribution, U (0 , Y − 1) .

Original: Speciﬁcally, the background generator G bg synthesizes a background image ˆ x bg ∈ R 3 × H × W solely from the random noise z , and the foreground generator G fg synthesizes a foreground mask ˆ m ∈ R 1 × H × W and an object image ˆ t ∈ R 3 × H × W using both z and c .
Revised: Speciﬁcally, the background generator G bg synthesizes a background image ˆ x bg ∈ R 3 × H × W solely from the random noise z , and the foreground generator G fg synthesizes a foreground mask ˆ m ∈ R 1 × H × W and an object image ˆ t ∈ R 3 × H × W using both z and c .

Original: To model foreground variations, we convert an 1-hot latent code c to a variable c (cid:48)
Revised: To model foreground variations, we convert an 1-hot latent code c to a variable c (cid:48)

Original: ∈
Revised: ∈

Original: R
Revised: R

Original: d c whose value is sampled from the gaussian distribution N ( µ c , σ c ) where the mean µ c and diagonal covariance matrix σ c are computed according to the original code c (Zhang et al., 2017).
Revised: d c whose value is sampled from the gaussian distribution N ( µ c , σ c ) where the mean µ c and diagonal covariance matrix σ c are computed according to the original code c (Zhang et al., 2017).

Original: The ﬁnal image ˆ x ∈ R 3 × H × W is the composition of generated image components summed by the hadamard product, as described in
Revised: The ﬁnal image ˆ x ∈ R 3 × H × W is the composition of generated image components summed by the hadamard product, as described in

Original: Fig.
Revised: Fig.

Original: To achieve a fully unsupervised method, we leverage the scene decomposition method proposed by
Revised: To achieve a fully unsupervised method, we leverage the scene decomposition method proposed by

Original: PerturbGAN (Bielski & Favaro, 2019).
Revised: PerturbGAN (Bielski & Favaro, 2019).

Original: In speciﬁc, scene decomposition is triggered by perturbing foreground components ˆ m and ˆ t with the random afﬁne transformation matrix T θ right before the ﬁnal image composition.
Revised: In speciﬁc, scene decomposition is triggered by perturbing foreground components ˆ m and ˆ t with the random afﬁne transformation matrix T θ right before the ﬁnal image composition.

Original: The parameters θ of the random matrix T θ include a rotation angle, scaling factor and translation distance, and they are all randomly sampled from the uniform distributions with predeﬁned value ranges.
Revised: The parameters θ of the random matrix T θ include a rotation angle, scaling factor and translation distance, and they are all randomly sampled from the uniform distributions with predeﬁned value ranges.

Original: In summation, the image generation process is described as below:
Revised: In summation, the image generation process is described as below:

Original: [Equation]
Revised: [Equation]

Original: 3. C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING
Revised: 3. C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING

Original: We assume that data would be well clustered when i) they form explicitly discernable cluster boundaries in the embedding space H , and ii)
Revised: We assume that data would be well clustered when i) they form explicitly discernable cluster boundaries in the embedding space H , and ii)

Original: each cluster centroid l y ∈ R
Revised: each cluster centroid l y ∈ R

Original: d h condenses distinct semantic features.
Revised: d h condenses distinct semantic features.

Original: This is the exact space that the discriminator of InfoGAN aims to approach when the latent code c is sampled from the uniform categorical distribution, U (0 , Y − 1) .
Revised: This is the exact space that the discriminator of InfoGAN aims to approach when the latent code c is sampled from the uniform categorical distribution, U (0 , Y − 1) .

Original: However, since it jointly optimizes the adversarial loss, the model has a possibility of falling into the mode collapse phenomenon where the generator G covers only a few conﬁdent modes (classes) to easily deceive the discriminator D .
Revised: However, since it jointly optimizes the adversarial loss, the model has a possibility of falling into the mode collapse phenomenon where the generator G covers only a few conﬁdent modes (classes) to easily deceive the discriminator D .

Original: This is the reason why InfoGAN lacks the ability of inferring clusters on real image datasets.
Revised: This is the reason why InfoGAN lacks the ability of inferring clusters on real image datasets.

Original: To handle this problem, we propose a new form of an auxiliary probability Q ( c | ˆ x ) that represents the mutual information between the latent code c and image observation ˆ x .
Revised: To handle this problem, we propose a new form of an auxiliary probability Q ( c | ˆ x ) that represents the mutual information between the latent code c and image observation ˆ x .

Original: Let us now describe the objective functions with mathematical deﬁnitions.
Revised: Let us now describe the objective functions with mathematical deﬁnitions.

Original: The discriminator D aims to learn an adversarial feature r ∈ R for the image authenticity discrimination, and a semantic feature h ∈ R d h for optimizing the information-theoretic regularization.
Revised: The discriminator D aims to learn an adversarial feature r ∈ R for the image authenticity discrimination, and a semantic feature h ∈ R d h for optimizing the information-theoretic regularization.

Original: The features are encoded from separate branches ψ rx and ψ hx , which were split at the end of the base encoder of the discriminator
Revised: The features are encoded from separate branches ψ rx and ψ hx , which were split at the end of the base encoder of the discriminator

Original: D base .
Revised: D base .

Original: The adversarial feature r is learnt with the hinge loss as represented in the equations below:
Revised: The adversarial feature r is learnt with the hinge loss as represented in the equations below:

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, the semantic feature h can be considered as a logit in a classiﬁcation model.
Revised: Meanwhile, the semantic feature h can be considered as a logit in a classiﬁcation model.

Original: In a supervised setting where ground-truth class labels are given, data mapped into an embedding space gradually form discernable cluster boundaries as the training proceeds, similar to the one in Fig.(b).
Revised: In a supervised setting where ground-truth class labels are given, data mapped into an embedding space gradually form discernable cluster boundaries as the training proceeds, similar to the one in Fig.(b).

Original: However, in an unsupervised setting, it is difﬁcult to attain such space simply by maximizing the mutual information between the latent code c and its observation ˆ x for the aforementioned reasons.
Revised: However, in an unsupervised setting, it is difﬁcult to attain such space simply by maximizing the mutual information between the latent code c and its observation ˆ x for the aforementioned reasons.

Original: Our solution is to formulate an auxiliary probability Q ( c | ˆ x ) in the form of contrastive loss.
Revised: Our solution is to formulate an auxiliary probability Q ( c | ˆ x ) in the form of contrastive loss.

Original: We map the latent code c onto the embedding space H with a simple linear layer ψ c , and let it act as a cluster centroid ( l ) that pulls semantic features of images ( h ) that were generated from its speciﬁc value.
Revised: We map the latent code c onto the embedding space H with a simple linear layer ψ c , and let it act as a cluster centroid ( l ) that pulls semantic features of images ( h ) that were generated from its speciﬁc value.

Original: By setting the relations with other latent codes as negative pairs, cluster centroids are set to be pushing each others and form distinguishable boundaries in the embedding space.
Revised: By setting the relations with other latent codes as negative pairs, cluster centroids are set to be pushing each others and form distinguishable boundaries in the embedding space.

Original: Speciﬁcally, we expect the semantic feature h to be mapped to the k -th cluster centroid, where k is the index of its corresponding latent code c that makes c k = 1 , while distances to other centroids l y are far enough to maximize the mutual information of the two.
Revised: Speciﬁcally, we expect the semantic feature h to be mapped to the k -th cluster centroid, where k is the index of its corresponding latent code c that makes c k = 1 , while distances to other centroids l y are far enough to maximize the mutual information of the two.

Original: To enhance the robustness of the model and assist the scene decomposition learning, we additionally maximize the mutual information between the latent code c and masked foreground images ˆ x fg .
Revised: To enhance the robustness of the model and assist the scene decomposition learning, we additionally maximize the mutual information between the latent code c and masked foreground images ˆ x fg .

Original: To sum up, the proposed information-theoretic objectives are deﬁned as follows:
Revised: To sum up, the proposed information-theoretic objectives are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where sim( a, b ) is the cosine distance between vectors a and b , y is an index of a cluster, and τ is the temperature of the softmax function.
Revised: where sim( a, b ) is the cosine distance between vectors a and b , y is an index of a cluster, and τ is the temperature of the softmax function.

Original: I Y denotes the identity matrix with the size of Y × Y .
Revised: I Y denotes the identity matrix with the size of Y × Y .

Original: Additional Regularizations Following the prior works (Ji et al., 2019; Van Gansbeke et al., 2020), we adopt the overclustering strategy to help the model learn more expressive feature set.
Revised: Additional Regularizations Following the prior works (Ji et al., 2019; Van Gansbeke et al., 2020), we adopt the overclustering strategy to help the model learn more expressive feature set.

Original: We also regularize the prediction on a real image p ∈ R Y by minimizing its entropy H ( p ) to promote each data to be mapped to only one cluster id with high conﬁdence, along with the minimization of the
Revised: We also regularize the prediction on a real image p ∈ R Y by minimizing its entropy H ( p ) to promote each data to be mapped to only one cluster id with high conﬁdence, along with the minimization of the

Original: KL divergence between batch-wisely averaged prediction ¯p and the uniform distribution u , that is for avoiding a degenerated solution where only a few clusters are overly allocated.
Revised: KL divergence between batch-wisely averaged prediction ¯p and the uniform distribution u , that is for avoiding a degenerated solution where only a few clusters are overly allocated.

Original: Furthermore, we optimize the contrastive loss (Chen et al., 2020) deﬁned for real image features h to assist the semantic feature learning.
Revised: Furthermore, we optimize the contrastive loss (Chen et al., 2020) deﬁned for real image features h to assist the semantic feature learning.

Original: To summarize, the regularizations on real images are as follows:
Revised: To summarize, the regularizations on real images are as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We also enforce a foreground mask ˆ m to be more like a hard mask and take up a reasonable portion of a scene by employing below regularization function.
Revised: We also enforce a foreground mask ˆ m to be more like a hard mask and take up a reasonable portion of a scene by employing below regularization function.

Original: [Equation]
Revised: [Equation]

Original: In sum, C3-GAN is trained alternately optimizing the following objective functions for D and G :
Revised: In sum, C3-GAN is trained alternately optimizing the following objective functions for D and G :

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETTING
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETTING

Original: Datasets.
Revised: Datasets.

Original: We tested our method on 4 datasets that consist of single object images.
Revised: We tested our method on 4 datasets that consist of single object images.

Original: i) CUB
Revised: i) CUB

Original: (Wah et al., 2011): 5,994 training and 5,794 test images of 200 bird species.
Revised: (Wah et al., 2011): 5,994 training and 5,794 test images of 200 bird species.

Original: ii)
Revised: ii)

Original: Stanford Cars (Krause et al., 2013): 8,144 training and 8,041 test images of 196 car models.
Revised: Stanford Cars (Krause et al., 2013): 8,144 training and 8,041 test images of 196 car models.

Original: iii)
Revised: iii)

Original: Stanford Dogs (Khosla et al., 2011): 12,000 training and 8,580 test images of 120 dog species.
Revised: Stanford Dogs (Khosla et al., 2011): 12,000 training and 8,580 test images of 120 dog species.

Original: iv) Oxford Flower (Nilsback & Zisserman, 2008): 2,040 training and 6,149 test images of 102 ﬂower categories.
Revised: iv) Oxford Flower (Nilsback & Zisserman, 2008): 2,040 training and 6,149 test images of 102 ﬂower categories.

Original: Due to the small number of training images, all models for CUB and Oxford Flower datasets were trained with the entire dataset as the prior works did (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020).
Revised: Due to the small number of training images, all models for CUB and Oxford Flower datasets were trained with the entire dataset as the prior works did (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020).

Original: Implementation Details.
Revised: Implementation Details.

Original: The weights of the loss terms ( λ 0 , λ 1 , λ 2 , λ 3 , λ 4 ) are set as (5, 1, 1, 0.1, 1), and the temperature τ is set as 0.1.
Revised: The weights of the loss terms ( λ 0 , λ 1 , λ 2 , λ 3 , λ 4 ) are set as (5, 1, 1, 0.1, 1), and the temperature τ is set as 0.1.

Original: We utilized Adam optimizer of which learning rate is 0.
Revised: We utilized Adam optimizer of which learning rate is 0.

Original: and values of momentum coefﬁcients are (0.5, 0.999).
Revised: and values of momentum coefﬁcients are (0.5, 0.999).

Original: The architectural speciﬁcation of C3-GAN and the values of hyperparameters, such as the parameter ranges of the random afﬁne transformation and the number of clusters Y can be found in Appendix A.1 and Appendix A.2, respectively.
Revised: The architectural speciﬁcation of C3-GAN and the values of hyperparameters, such as the parameter ranges of the random afﬁne transformation and the number of clusters Y can be found in Appendix A.1 and Appendix A.2, respectively.

Original: Baselines.
Revised: Baselines.

Original: We ﬁrst compared C3-GAN with IIC (Ji et al., 2019) in order to emphasize the difference between the coarse-grained and ﬁne-grained class clustering task, since it achieved decent performance with sobel ﬁltered images where color and texture informations are discarded.
Revised: We ﬁrst compared C3-GAN with IIC (Ji et al., 2019) in order to emphasize the difference between the coarse-grained and ﬁne-grained class clustering task, since it achieved decent performance with sobel ﬁltered images where color and texture informations are discarded.

Original: We also experimented with simple baseline of SimCLR+K-Means (Chen et al., 2020) to claim that optimizing instance discrimination task is not enough for ﬁne-grained feature learning.
Revised: We also experimented with simple baseline of SimCLR+K-Means (Chen et al., 2020) to claim that optimizing instance discrimination task is not enough for ﬁne-grained feature learning.

Original: SCAN (Van Gansbeke et al., 2020), the work that shows a remarkable coarse-grained class clustering performance, is also compared to investigate whether the method of inducing consistent predictions for all nearest neighbors would be helpful for the ﬁne-grained class clustering task as well.
Revised: SCAN (Van Gansbeke et al., 2020), the work that shows a remarkable coarse-grained class clustering performance, is also compared to investigate whether the method of inducing consistent predictions for all nearest neighbors would be helpful for the ﬁne-grained class clustering task as well.

Original: Finally, we compared with the methods that learn hierarchically ﬁne-grained feature representations, such as FineGAN (Singh et al., 2019),
Revised: Finally, we compared with the methods that learn hierarchically ﬁne-grained feature representations, such as FineGAN (Singh et al., 2019),

Original: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020), including their base model InfoGAN (Chen et al., 2016), to claim the efﬁcacy of the proposed formulation that is based on the contrastive loss.
Revised: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020), including their base model InfoGAN (Chen et al., 2016), to claim the efﬁcacy of the proposed formulation that is based on the contrastive loss.

Original: We mainly compared with the unsupervised version of them that is trained with pseudo-labeled bounding boxes which assign edges of real images as background region.
Revised: We mainly compared with the unsupervised version of them that is trained with pseudo-labeled bounding boxes which assign edges of real images as background region.

Original: 4. F INE - GRAINED C LASS C LUSTERING R ESULTS
Revised: 4. F INE - GRAINED C LASS C LUSTERING R ESULTS

Original: Quantitative results.
Revised: Quantitative results.

Original: We evaluated clustering performance with two metrics: Accuracy (Acc) and
Revised: We evaluated clustering performance with two metrics: Accuracy (Acc) and

Original: Normalized Mutual Information (NMI).
Revised: Normalized Mutual Information (NMI).

Original: The score of accuracy is calculated following the optimal mapping between cluster indices and real classes inferred by Hungarian algorithm (Kuhn & Yaw, 1955) for a given contingency table.
Revised: The score of accuracy is calculated following the optimal mapping between cluster indices and real classes inferred by Hungarian algorithm (Kuhn & Yaw, 1955) for a given contingency table.

Original: We presented the results in Table 1.
Revised: We presented the results in Table 1.

Original: As it can be seen, C3-GAN outperforms other methods by remarkable margins in terms of Acc on all datasets.
Revised: As it can be seen, C3-GAN outperforms other methods by remarkable margins in terms of Acc on all datasets.

Original: Our method presents better or comparable performance in terms of NMI scroes as well.
Revised: Our method presents better or comparable performance in terms of NMI scroes as well.

Original: The results of IIC underline that understanding only structural characteristic is not enough for the ﬁne-grained class clustering task.
Revised: The results of IIC underline that understanding only structural characteristic is not enough for the ﬁne-grained class clustering task.

Original: From the results of SCAN (2nd rank) and MixNMatch (3rd rank), we can conjecture that both requiring local disentanglement of the embedding space and extracting foreground features via scene decomposition learning can be helpful for improving the ﬁne-grained class clustering performance.
Revised: From the results of SCAN (2nd rank) and MixNMatch (3rd rank), we can conjecture that both requiring local disentanglement of the embedding space and extracting foreground features via scene decomposition learning can be helpful for improving the ﬁne-grained class clustering performance.

Original: However, it is difﬁcult to determine whether the GAN-based methods or the SSLbased methods are better because both approaches have similar score ranges for all metrics and display globally entangled feature spaces as shown in Fig.
Revised: However, it is difﬁcult to determine whether the GAN-based methods or the SSLbased methods are better because both approaches have similar score ranges for all metrics and display globally entangled feature spaces as shown in Fig.

Original: 3 (a) and (b).
Revised: 3 (a) and (b).

Original: Since C3-GAN resembles
Revised: Since C3-GAN resembles

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: MixNMatch to some extent, its state-of-the-art scores can be partially explained by the fact that it learns decomposed foreground features, but the embedding space of C3-GAN visualized in Fig.
Revised: MixNMatch to some extent, its state-of-the-art scores can be partially explained by the fact that it learns decomposed foreground features, but the embedding space of C3-GAN visualized in Fig.

Original: 3 (c) implies that the performance of it has been further improved from our formulation of Q ( c | ˆ x ) that enforces clusters to be distinctively distributed in the embedding space.
Revised: 3 (c) implies that the performance of it has been further improved from our formulation of Q ( c | ˆ x ) that enforces clusters to be distinctively distributed in the embedding space.

Original: We additionally presented the results of qualitative analysis on clustered images inferred by the top three methods in Appendix A.3, which demonstrate the excellence of our method compared to prior works.
Revised: We additionally presented the results of qualitative analysis on clustered images inferred by the top three methods in Appendix A.3, which demonstrate the excellence of our method compared to prior works.

Original: Ablation Study.
Revised: Ablation Study.

Original: We conducted an ablation study regarding three factors: i)
Revised: We conducted an ablation study regarding three factors: i)

Original: Overclustering, ii)
Revised: Overclustering, ii)

Original: Foreground perturbation, and iii)
Revised: Foreground perturbation, and iii)

Original: The information-theoretic regularization based on the contrastive loss.
Revised: The information-theoretic regularization based on the contrastive loss.

Original: We investigated clustering performance of C3-GAN on CUB dataset by removing each factor.
Revised: We investigated clustering performance of C3-GAN on CUB dataset by removing each factor.

Original: The results can be seen in Table 2.
Revised: The results can be seen in Table 2.

Original: The largest performance degradation was caused by changing the formulation of the information-theoretic regularization to a trivial softmax function.
Revised: The largest performance degradation was caused by changing the formulation of the information-theoretic regularization to a trivial softmax function.

Original: This result implies that clustering performance has a high correlation with the degree of separability of clusters, which was signiﬁcantly improved by our method.
Revised: This result implies that clustering performance has a high correlation with the degree of separability of clusters, which was signiﬁcantly improved by our method.

Original: Regarding the result of ii), we observed that C3-GAN fails to decompose a scene when the foreground perturbation was not implemented.
Revised: Regarding the result of ii), we observed that C3-GAN fails to decompose a scene when the foreground perturbation was not implemented.

Original: This degenerated result suggests that the method of extracting only foreground features has a non-negligible amount of inﬂuence in clustering performance.
Revised: This degenerated result suggests that the method of extracting only foreground features has a non-negligible amount of inﬂuence in clustering performance.

Original: Lastly, the result of i) implies that the overclustering strategy is also helpful for boosting performance since it allows a larger capacity for model to learn more expressive feature set.
Revised: Lastly, the result of i) implies that the overclustering strategy is also helpful for boosting performance since it allows a larger capacity for model to learn more expressive feature set.

Original: The additional analysis on hyperparameters can be found in Appendix A.2.
Revised: The additional analysis on hyperparameters can be found in Appendix A.2.

Original: [Table]
Revised: [Table]

Original: i) – Overclustering
Revised: i) – Overclustering

Original: [Table]
Revised: [Table]

Original: ii)
Revised: ii)

Original: – Foreground perturbation
Revised: – Foreground perturbation

Original: [Table]
Revised: [Table]

Original: iii
Revised: iii

Original: [Table]
Revised: [Table]

Original: theoretic
Revised: theoretic

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4. I MAGE G ENERATION R ESULTS Qualitative Results
Revised: 4. I MAGE G ENERATION R ESULTS Qualitative Results

Original: For qualitative evaluation, we analyzed the results on CUB dataset.
Revised: For qualitative evaluation, we analyzed the results on CUB dataset.

Original: In Fig.(a), we displayed synthesized images along with their decomposed scene elements.
Revised: In Fig.(a), we displayed synthesized images along with their decomposed scene elements.

Original: The images were generated with the cluster indices of real images that were predicted by the discriminator.
Revised: The images were generated with the cluster indices of real images that were predicted by the discriminator.

Original: Considering the consistency of foreground object features within the same class and the uniqueness of each class feature, we can assume that C3-GAN succeeded in learning the set of clusters in which each cluster represents its unique characteristic explicitly.
Revised: Considering the consistency of foreground object features within the same class and the uniqueness of each class feature, we can assume that C3-GAN succeeded in learning the set of clusters in which each cluster represents its unique characteristic explicitly.

Original: Further, to investigate the roles of the two input variables, c and z , we analyzed the change in the synthesized images when they were generated by ﬁxing one of the two and diversifying the other, as shown in Fig.
Revised: Further, to investigate the roles of the two input variables, c and z , we analyzed the change in the synthesized images when they were generated by ﬁxing one of the two and diversifying the other, as shown in Fig.

Original: 4 (b).
Revised: 4 (b).

Original: As it can be seen, if c is ﬁxed, all images depict the same bird species, but the pose of a bird and background vary depending on the value of z .
Revised: As it can be seen, if c is ﬁxed, all images depict the same bird species, but the pose of a bird and background vary depending on the value of z .

Original: Conversely, when generated under the condition of variable c and ﬁxed z , images of various bird species with the same pose and background were observed.
Revised: Conversely, when generated under the condition of variable c and ﬁxed z , images of various bird species with the same pose and background were observed.

Original: The results on other datasets show the same trend.
Revised: The results on other datasets show the same trend.

Original: Please refer to Appendix A.4.
Revised: Please refer to Appendix A.4.

Original: It is also notable that C3-GAN can effectively alleviate the mode collapse issue with the proposed method.
Revised: It is also notable that C3-GAN can effectively alleviate the mode collapse issue with the proposed method.

Original: Fig.
Revised: Fig.

Original: 5 present that only C3-GAN reﬂects intra-class deviations of various factors such as color, shape, and layout for the given class, while other baselines produce images only with the layout variation.
Revised: 5 present that only C3-GAN reﬂects intra-class deviations of various factors such as color, shape, and layout for the given class, while other baselines produce images only with the layout variation.

Original: We conjecture that this performance was achieved because the learnt embedding space of the discriminator better represents the real data distribution, allowing the generator to get quality
Revised: We conjecture that this performance was achieved because the learnt embedding space of the discriminator better represents the real data distribution, allowing the generator to get quality

Original: [Figure]
Revised: [Figure]

Original: Only C3-GAN generates images reﬂecting the intra -class color, shape, and layout variations of the real image set.
Revised: Only C3-GAN generates images reﬂecting the intra -class color, shape, and layout variations of the real image set.

Original: [Table]
Revised: [Table]

Original: signals for the adversarial learning.
Revised: signals for the adversarial learning.

Original: Additionally, we found that only C3-GAN succeeded in the scene decomposition learning when annotations were not utilized.
Revised: Additionally, we found that only C3-GAN succeeded in the scene decomposition learning when annotations were not utilized.

Original: In fact, when FineGAN and MixNMatch were trained without object bounding box annotations, they fell into degenerate solutions where the entire image is synthesized from the background generators.
Revised: In fact, when FineGAN and MixNMatch were trained without object bounding box annotations, they fell into degenerate solutions where the entire image is synthesized from the background generators.

Original: Quantitative Results.
Revised: Quantitative Results.

Original: We also quantitatively evaluated the image synthesis performance based on the scores of Frechet Inception Distance (FID) and Inception Score (IS).
Revised: We also quantitatively evaluated the image synthesis performance based on the scores of Frechet Inception Distance (FID) and Inception Score (IS).

Original: We also considered the scores of reverse KL which measures the distance between the predicted class distribution of real images and synthesized images.
Revised: We also considered the scores of reverse KL which measures the distance between the predicted class distribution of real images and synthesized images.

Original: We compared C3-GAN with GAN-based models that were trained without object bounding box annotations.
Revised: We compared C3-GAN with GAN-based models that were trained without object bounding box annotations.

Original: The scores of IS and reverse KL were measured with
Revised: The scores of IS and reverse KL were measured with

Original: Inception networks that were ﬁne-tuned on each dataset using the method of Cui et al.
Revised: Inception networks that were ﬁne-tuned on each dataset using the method of Cui et al.

Original: The results are displayed in Table 3.
Revised: The results are displayed in Table 3.

Original: C3-GAN presents the state-of-the-art or comparable performance in terms of all metrics.
Revised: C3-GAN presents the state-of-the-art or comparable performance in terms of all metrics.

Original: This result means that, in addition to the class clustering performance, C3-GAN has competitive performance in generating ﬁne-grained object images.
Revised: This result means that, in addition to the class clustering performance, C3-GAN has competitive performance in generating ﬁne-grained object images.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this study, we proposed a new GAN-based method, C3-GAN, for unsupervised ﬁne-grained class clustering which is more challenging and less explored than a coarse-grained clustering task.
Revised: In this study, we proposed a new GAN-based method, C3-GAN, for unsupervised ﬁne-grained class clustering which is more challenging and less explored than a coarse-grained clustering task.

Original: To improve the ﬁne-grained class clustering performance, we formulate the information-theoretic regularization based on the contrastive loss.
Revised: To improve the ﬁne-grained class clustering performance, we formulate the information-theoretic regularization based on the contrastive loss.

Original: Also, a scene decomposition-based approach is incorporated into the framework to enforce the model to learn features focusing on a foreground object.
Revised: Also, a scene decomposition-based approach is incorporated into the framework to enforce the model to learn features focusing on a foreground object.

Original: Extensive experiments show that our C3-GAN not only outperforms previous ﬁne-grained clustering methods but also synthesizes ﬁne-grained object images with comparable quality, while alleviating mode collapse that previous state-of-the-art GAN methods have been suffering from.
Revised: Extensive experiments show that our C3-GAN not only outperforms previous ﬁne-grained clustering methods but also synthesizes ﬁne-grained object images with comparable quality, while alleviating mode collapse that previous state-of-the-art GAN methods have been suffering from.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Remarkable advancement of generative models has played a crucial role as AI tools for text (Brown et al., 2020;
Revised: Remarkable advancement of generative models has played a crucial role as AI tools for text (Brown et al., 2020;

Original: Kim et al., 2021), image (Karras et al., 2020; Choi et al., 2020b), audio (Sisman et al., 2020), and multimodal content generation (Tian et al., 2021; Yan et al., 2021).
Revised: Kim et al., 2021), image (Karras et al., 2020; Choi et al., 2020b), audio (Sisman et al., 2020), and multimodal content generation (Tian et al., 2021; Yan et al., 2021).

Original: However, as its side effect, many malicious applications have been reported as well, thus leading to severe societal problems, such as deepfake (Dolhansky et al., 2020), fake news (Lazer et al., 2018) and generation biased by training data (Gupta et al., 2021).
Revised: However, as its side effect, many malicious applications have been reported as well, thus leading to severe societal problems, such as deepfake (Dolhansky et al., 2020), fake news (Lazer et al., 2018) and generation biased by training data (Gupta et al., 2021).

Original: Our work might be an extension of the harmful effects of generative models.
Revised: Our work might be an extension of the harmful effects of generative models.

Original: On the contrary, generative models can be a solution to alleviate these side effects via data augmentation (Yang et al., 2019; Choi et al., 2020a).
Revised: On the contrary, generative models can be a solution to alleviate these side effects via data augmentation (Yang et al., 2019; Choi et al., 2020a).

Original: In particular, our method can contribute to alleviating data bias and ﬁne-grained class imbalance.
Revised: In particular, our method can contribute to alleviating data bias and ﬁne-grained class imbalance.

Original: Therefore, many endless efforts are required to make these powerful generative models beneﬁt humans.
Revised: Therefore, many endless efforts are required to make these powerful generative models beneﬁt humans.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was experimented on the NAVER Smart Machine Learning (NSML) platform (Sung et al., 2017; Kim et al., 2018).
Revised: This work was experimented on the NAVER Smart Machine Learning (NSML) platform (Sung et al., 2017; Kim et al., 2018).

Original: We are especially grateful to Jun-Yan Zhu and NAVER AI Lab researchers for their constructive comments.
Revised: We are especially grateful to Jun-Yan Zhu and NAVER AI Lab researchers for their constructive comments.

Original: A A PPENDIX A. N ETWORK A RCHITECTURE
Revised: A A PPENDIX A. N ETWORK A RCHITECTURE

Original: We present detailed description of the architecture of C3-GAN in Table 4 and Table 5.
Revised: We present detailed description of the architecture of C3-GAN in Table 4 and Table 5.

Original: The colored dots ( • , • , • ) are for indicating that the outputs of earlier layers are used as the inputs of later layers.
Revised: The colored dots ( • , • , • ) are for indicating that the outputs of earlier layers are used as the inputs of later layers.

Original: The training was done with 2 NVIDIA-V100 GPUs, and we optimized a model until the convergence of the FID score
Revised: The training was done with 2 NVIDIA-V100 GPUs, and we optimized a model until the convergence of the FID score

Original: [Table] [Table]
Revised: [Table] [Table]

Original: • Conv( a , b ) : 2D Convolution layer with kernel size a and stride b • GLU : Gated Linear Units
Revised: • Conv( a , b ) : 2D Convolution layer with kernel size a and stride b • GLU : Gated Linear Units

Original: A. A DDITIONAL A BLATION S TUDY Random afﬁne transformation.
Revised: A. A DDITIONAL A BLATION S TUDY Random afﬁne transformation.

Original: We experimented with two types of purtubing policy, a weak and strong random afﬁne transformation.
Revised: We experimented with two types of purtubing policy, a weak and strong random afﬁne transformation.

Original: The detailed value ranges of their parameters are presented in Table 6.
Revised: The detailed value ranges of their parameters are presented in Table 6.

Original: The type of perturbation used for each dataset was determined by manually checking randomly perturbed real images.
Revised: The type of perturbation used for each dataset was determined by manually checking randomly perturbed real images.

Original: The strong perturbation was only used for CUB dataset, and the rest datasets used the weak perturbation.
Revised: The strong perturbation was only used for CUB dataset, and the rest datasets used the weak perturbation.

Original: [Table]
Revised: [Table]

Original: Overclustering.
Revised: Overclustering.

Original: We use four datasets for the evaluation, including CUB, Standford Cars, Stanford
Revised: We use four datasets for the evaluation, including CUB, Standford Cars, Stanford

Original: Dogs and Oxford Flower.
Revised: Dogs and Oxford Flower.

Original: As we described in 3.2, we employ the overclustering policy which is to set the number of clusters to be multiple times of the actual number of classes.
Revised: As we described in 3.2, we employ the overclustering policy which is to set the number of clusters to be multiple times of the actual number of classes.

Original: To ﬁnd the optimal setting, we compared the results of C3-GAN by setting the number of clusters as 1, 2, and 3 times the actual number of classes.
Revised: To ﬁnd the optimal setting, we compared the results of C3-GAN by setting the number of clusters as 1, 2, and 3 times the actual number of classes.

Original: The cluster numbers for each dataset were determined based on the results in Table 7.
Revised: The cluster numbers for each dataset were determined based on the results in Table 7.

Original: We found that the performance generally tends to improve as the number of clusters increases.
Revised: We found that the performance generally tends to improve as the number of clusters increases.

Original: This implies that overclustering is indeed helpful for the expressive feature learning.
Revised: This implies that overclustering is indeed helpful for the expressive feature learning.

Original: To further investigate the results when we are not aware of the actual number of classes, we additionally conducted all experiments by setting the number of clusters as 100 (underclustering) or 500 (overclustering).
Revised: To further investigate the results when we are not aware of the actual number of classes, we additionally conducted all experiments by setting the number of clusters as 100 (underclustering) or 500 (overclustering).

Original: The results are represented in Table 8.
Revised: The results are represented in Table 8.

Original: For the underclustering setting, we only report the scores of NMI, since some classes have no matching cluster indices.
Revised: For the underclustering setting, we only report the scores of NMI, since some classes have no matching cluster indices.

Original: Despite the less promising performance of the underclustering case, this result implies that our method can be applied to any datasets whose cluster size is not available, if we set a large enough number of clusters.[Equation]
Revised: Despite the less promising performance of the underclustering case, this result implies that our method can be applied to any datasets whose cluster size is not available, if we set a large enough number of clusters.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. Q UALITATIVE A NALYSIS OF C LUSTERING R ESULTS
Revised: A. Q UALITATIVE A NALYSIS OF C LUSTERING R ESULTS

Original: We present the clustered images of CUB and Oxford Flowers datasets, along with the results of
Revised: We present the clustered images of CUB and Oxford Flowers datasets, along with the results of

Original: MixNMatch and SCAN in Figs.
Revised: MixNMatch and SCAN in Figs.

Original: 6 and 7.
Revised: 6 and 7.

Original: From these results, we can assume that our method is better at clustering compared to the baseline methods.
Revised: From these results, we can assume that our method is better at clustering compared to the baseline methods.

Original: It is worth noting that even incorrectly assigned images look very similar with the given bird species for C3-GAN, while the results of other methods contain objects that are quite deviated from the condition.
Revised: It is worth noting that even incorrectly assigned images look very similar with the given bird species for C3-GAN, while the results of other methods contain objects that are quite deviated from the condition.

Original: We could observe the similar trend for
Revised: We could observe the similar trend for

Original: Oxford Flowers dataset, as it is shown in Fig.
Revised: Oxford Flowers dataset, as it is shown in Fig.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. A DDITIONAL IMAGE GENERATION RESULTS
Revised: A. A DDITIONAL IMAGE GENERATION RESULTS

Original: A.4.
Revised: A.4.

Original: CUB
Revised: CUB

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background Background
Revised: Background Background

Original: Background +
Revised: Background +

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: Fixed c and variable z
Revised: Fixed c and variable z

Original: Varible c and ﬁxed z
Revised: Varible c and ﬁxed z

Original: A.4.
Revised: A.4.

Original: S TANFORD C ARS
Revised: S TANFORD C ARS

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background
Revised: Background

Original: Background
Revised: Background

Original: Background + Foreground + Foreground + Foreground
Revised: Background + Foreground + Foreground + Foreground

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: z
Revised: z

Original: Varible
Revised: Varible

Original: [Table]
Revised: [Table]

Original: and ﬁxed z
Revised: and ﬁxed z

Original: A.4. S TANFORD D OGS
Revised: A.4. S TANFORD D OGS

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background
Revised: Background

Original: Background
Revised: Background

Original: Background + Foreground + Foreground + Foreground
Revised: Background + Foreground + Foreground + Foreground

Original: Fixed c and variable z
Revised: Fixed c and variable z

Original: Varible c and ﬁxed z
Revised: Varible c and ﬁxed z

Original: A.4. 
Revised: A.4.

Original: O
Revised: O

Original: XFORD
Revised: XFORD

Original: F
Revised: F

Original: LOWER
Revised: LOWER

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background
Revised: Background

Original: Background
Revised: Background

Original: Background + Foreground + Foreground + Foreground
Revised: Background + Foreground + Foreground + Foreground

Original: Fixed c and variable z Varible c and ﬁxed z
Revised: Fixed c and variable z Varible c and ﬁxed z