Original: C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING VIA G ENERATIVE A DVERSARIAL N ETWORKS
Revised: C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING VIA G ENERATIVE A DVERSARIAL N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised ﬁne-grained class clustering is  practical yet challenging  due to the difﬁculty of feature representations learning of subtle object details.
Revised: Unsupervised ﬁne-grained class clustering is a practical yet challenging task due to the difﬁculty of feature representations learning of subtle object details.

Original: We introduce C3-GAN, a method that leverages the categorical inference power of InfoGAN with contrastive learning.
Revised: We introduce C3-GAN, a method that leverages the categorical inference power of InfoGAN with contrastive learning.

Original: We aim to learn feature representations thatnot only maximize the mutual information between the latent code and its observation, but also encourage the data to form distinct cluster boundaries in the embedding space.
Revised: We aim to learn feature representations that encourage a dataset to form distinct cluster boundaries in the embedding space, while also maximizing the mutual information between the latent code and its image observation.

Original: Our approach is to train the discriminator optimizing the contrastive loss, wherethe image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs, then we utilize it for clustering images.
Revised: Our approach is to train a discriminator, which is also used for inferring clusters, to optimize the contrastive loss, where image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs.

Original: For this formulation, we map thelatent code, which is the input of the generator,  to the embedding space of the discriminator and let them act as a cluster centroid.
Revised: Speciﬁcally, we map the input of a generator, which was sampled from the categorical distribution, to the embedding space of the discriminator and let them act as a cluster centroid.

Original: In this way, C3-GAN achieved to learn a clustering-friendly embedding space where each cluster is distinctively separated.
Revised: In this way, C3-GAN succeeded in learning a clustering-friendly embedding space where each cluster is distinctively separable.

Original: Experiments show that C3-GAN achieved state-of-the-art clustering performances on four ﬁne-grained benchmark datasets, while also alleviating the mode collapse phenomenon. 
Revised: Experimental results show that C3-GAN achieved state-of-the-art clustering performances on four ﬁne-grained image datasets, while also alleviating the mode collapse phenomenon. Code is available at https://github.com/naver-ai/c3-gan .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unsupervised ﬁne-grained class clustering is to group images of very similar objects (Benny & Wolf, 2020).
Revised: Unsupervised ﬁne-grained class clustering is a task that classiﬁes images of very similar objects (Benny & Wolf, 2020).

Original: While a line of works based on multiview-based self-supervised learning (SSL) methods (He et al., 2020; Chen et al., 2020; Grill et al., 2020) shows promising results in the conventional coarsegrained class clustering  (Van Gansbeke et al., 2020) task, it is difﬁcult for a ﬁne-grained  clustering task to beneﬁt from these methods for two reasons.
Revised: While a line of works based on multiview-based self-supervised learning (SSL) methods (He et al., 2020; Chen et al., 2020; Grill et al., 2020) shows promising results on a conventional coarse-grained class clustering task (Van Gansbeke et al., 2020), it is difﬁcult for a ﬁne-grained class clustering task to beneﬁt from these methods for two reasons.

Original: First, though it is more challenging and ambiguous for ﬁnding the distinction between ﬁne-grained classes, datasets  of the ﬁne-grained object images are difﬁcult to be large-scale.
Revised: First, though it is more challenging and ambiguous for ﬁnding distinctions between ﬁne-grained classes, datasets for this type of task are difﬁcult to be large-scale.

Original:  Figure 1 visually represents that ﬁnding distinctions between these classes are more difﬁcult compared to the ones of the coarse-grained object classes.
Revised: We visualized this idea in Figure 1 which shows that ﬁnding distinctions between ﬁne-grained classes is more difﬁcult than doing so for coarse-grained object classes.

Original: Second, the augmentation process of these methods considerthe subtle changes in color or shape, that actually play an important role in sorting classes, as noisy factors.
Revised: Second, the augmentation processes in these methods consider subtle changes in color or shape, that actually play an important role in differentiating between classes, as noisy factors.

Original: Thus, it is required to ﬁnd an another approach for the ﬁne-grained class clustering task.
Revised: Thus, it is required to ﬁnd an another approach for a ﬁne-grained class clustering task.

Original: Generative adversarial networks (GAN) (Goodfellow et al., 2014) can be a solution as it is required to learn ﬁne details to generate realistic images.
Revised: Generative adversarial networks (GAN) (Goodfellow et al., 2014) can be a solution as it needs to learn ﬁne details to generate realistic images.

Original: The possible starting point can be InfoGAN (Chen et al., 2016), which has achieved unsupervised categorical inference on MNIST dataset (LeCun et al., 2010) by maximizing the mutual information between the latent code and its observation.
Revised: A possible starting point could be InfoGAN (Chen et al., 2016), that succeeded in unsupervised categorical inference on MNIST dataset (LeCun et al., 2010) by maximizing the mutual information between the latent code and its observation.

Original: The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes.
Revised: The only prior knowledge employed was the number of classes and the fact that the data is uniformly distributed over the classes.

Original: FineGAN (Singh et al., 2019) extends InfoGAN by integrating the scene decompositionlearning method into the framework, and learns three latent codes that all follow the uniform categorical distribution.
Revised: FineGAN (Singh et al., 2019) extends InfoGAN by integrating the scene decomposition method into the framework, and learns three latent codes for the hierarchical scene generation.

Original: Each code has a hierarchical relationship with each other, and is sequentially injected to the generator for a background, a super-class object, and a sub-class object image syntheses.
Revised: Each code, that is sampled from the independent uniform categorical distribution, is sequentially injected to multiple generators for a background, a super-class object, and a sub-class object image syntheses.

Original: FineGAN demonstrates that the learnt codes for foreground image generation can also be used for clustering real images into their ﬁne-grained classes and outperforms conventional coarse-grained class clustering methods.
Revised: FineGAN demonstrated that two latent codes for object image generations could be also utilized for clustering real images into their ﬁne-grained classes, outperforming conventional coarse-grained class clustering methods.

Original: This result implies that extracting onlythe foreground part is very helpful for the given task.
Revised: This result implies that extracting only foreground features is very helpful for the given task.

Original: However, FineGAN requires  bounding box annotations and additional training of classiﬁers, which hinders its applicability.
Revised: However, FineGAN requires object bounding box annotations and additional training of classiﬁers, which greatly hinder its applicability.

Original: Also, the method lacks the ability of learning the distribution of real images due tothe mode collapse phenomenon (Higgins et al., 2017).
Revised: Also, the method lacks the ability of learning the distribution of real images, due to

Original: [Figure]
Revised: [Figure]

Original: In this paper, we propose C onstrastive ﬁne-grained C lass C lustering GAN (C3-GAN) that deals with the aforementioned problems.
Revised: In this paper, we propose C onstrastive ﬁne-grained C lass C lustering GAN (C3-GAN) that deals with the aforementioned problems.

Original: We ﬁrst remove the reliance on human-annotated labels by adopting the method  inspired by PerturbGAN (Bielski & Favaro, 2019) for unsupervised scene decomposition learning.
Revised: We ﬁrst remove the reliance on human-annotated labels by adopting the method for unsupervised scene decomposition learning that is inspired by PerturbGAN (Bielski & Favaro, 2019).

Original: And, with the intuition that optimalclustering results would be achieved if clusters of image features areuniformly distributed and can be linearly separablefrom each other (Wang & Isola, 2020), we train the discriminator to optimize the contrastive loss (He et al., 2020), where the image-latent feature pairs that maximize the mutual information are set as positive pairs.
Revised: We then adopt contrastive learning method (van den Oord et al., 2018) for training a discriminator by deﬁning pairs of image-latent features that maximize the mutual information of the two as positive pairs and the rest as negative pairs. This is based on the intuition that optimal results would be obtained if cluster centroids are distributed in a way that each cluster can be linearly separable (Wang & Isola, 2020).

Original: In speciﬁc, we map thelatent code, which is the input of  the generator, onto the embedding space of the discriminator and let it act as a cluster centroid that pulls the feature of the image that has generated with this speciﬁc value.
Revised: In speciﬁc, we map the input of a generator that was sampled from the categorical distribution onto the embedding space of the discriminator, and let it act as a cluster centroid that pulls features of images that were generated with its speciﬁc value.

Original: By setting the relation with other latent codes as negative pairs, the clustercentroids would be pushing each others so that the embedding space to have the characteristic described earlier.
Revised: Since the relations with other latent values are set as negative pairs, each cluster would be placed farther away from each other by the nature of the contrastive loss.

Original: The induced embedding space of the discriminator is also helpful for image generation learning, as the generator is required to have the ability to synthesize certain number of object classes with their distinct characteristics.

Original: We have conducted experiments on four ﬁne-grained  datasets including CUB, Stanford Cars, Stanford Dogs and Oxford Flower, and show that our C3-GAN achieved state-of-the-art clustering performance on all the datasets while also alleviating the mode collapse issue.
Revised: We have conducted experiments on four ﬁne-grained image datasets including CUB, Stanford Cars, Stanford Dogs and Oxford Flower, and demonstrated the effectiveness of C3-GAN by showing that it achieves the state-of-the-art clustering performances on all datasets.

Revised: Moreover, the induced embedding space of the discriminator turned out to be also helpful for alleviating the mode collapse issue.

Revised: We conjecture that this is because the generator is additionally required to be able to synthesize a certain number of object classes with distinct characteristics.

Original: Our main contributions are summarized as three folds:
Revised: Our main contributions are summarized as follows:

Original: • We propose a novel form of  information-theoretic regularization using the contrastive loss. With the proposed formulation, we directly learn the cluster centroids that would lead the data to form distinct cluster boundariesin the embedding space without falling into a degenerated solution.
Revised: • We propose a novel form of the information-theoretic regularization to learn a clusteringfriendly embedding space that leads a dataset to form distinct cluster boundaries without falling into a degenerated solution.

Original: • By extracting foreground features via scene decomposition learning, our method achieves the state-of-the-art performance on the  ﬁne-grained class clustering while not requiring any human-annotated label.
Revised: With this method, our C3-GAN achieved the state-ofthe-art ﬁne-grained class clustering performances on four ﬁne-grained image datasets. • By adopting scene decomposition learning method that does not require any humanannotated labels, our method can be applied to more diverse datasets.

Original: • Our model is capable of generating images with the controllability over the object classes, and also good at alleviating the mode collapse issue of GAN.
Revised: • Our method of training a discriminator is not only suitable for class clustering task, but also good at alleviating the mode collapse issue of GANs.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Unsupervised Clustering.
Revised: Unsupervised Clustering.

Original: Unsupervised clustering methods can be mainly categorized into  information theory-based and Expectation–Maximization (EM)-based approaches.
Revised: Unsupervised clustering methods can be mainly categorized into the information theory-based and Expectation–Maximization(EM)-based approaches.

Original: The ﬁrst approach aims to maximize  mutual information betweengroups of original images and augmented images in an end-to-end fashion (Ji et al., 2019; Zhong et al., 2020).
Revised: The ﬁrst approach aims to maximize the mutual information between original images and their augmentations to train a model in an end-to-end fashion (Ji et al., 2019; Zhong et al., 2020).

Original: To enhance the performance, IIC (Ji et al., 2019) additionally trains the auxiliary branch using an unlabeled large dataset, and DRC (Zhong et al., 2020) optimizes the contrastive loss on logit features for reducing the intra-class feature variation.
Revised: To enhance the performance, IIC (Ji et al., 2019) additionally trains the auxiliary branch using an unlabeled large dataset, and DRC (Zhong et al., 2020) optimizes the contrastive loss on logit features for reducing the intra-class feature variation.

Original: EM-based methods decouple the cluster assigning process (E-step) and the feature representations learning process (M-step) to learn more robustfeature representations.
Revised: EM-based methods decouple the cluster assignment process (E-step) and the feature representations learning process (M-step) to learn more robust representations.

Original: Speciﬁcally, the feature representations are learned by not only ﬁtting the proto clustering results estimated in the expectation step but also optimizing particular pretext tasks.
Revised: Speciﬁcally, the feature representations are learnt by not only ﬁtting the proto-clustering results estimated in the expectation step, but also optimizing particular pretext tasks.

Original: The clusters are  inferred by k -means clustering (Xie et al., 2016; Caron et al., 2018; Liu et al., 2020; Li et al., 2021) or training of an additional encoder (Dizaji et al., 2017).
Revised: The clusters are either inferred by k -means clustering (Xie et al., 2016; Caron et al., 2018; Liu et al., 2020; Li et al., 2021) or training of an additional encoder (Dizaji et al., 2017).

Original: However,  these methods suffer from the uneven allocations issue of k -means clustering, which could result in a degenerate solution.
Revised: However, many of these methods suffer from an uneven allocation issue of k -means clustering, which could result in a degenerate solution.

Original: For this reason, SCAN (Van Gansbeke et al., 2020) proposes a novel objective function that doesn’t require  k -means clustering.
Revised: For this reason, SCAN (Van Gansbeke et al., 2020) proposes the novel objective function that does not require the k -means clustering process.

Original: Based on the observation that K nearest neighbors of each feature  belong to the same ground-truth clusters with the high probability, it trains a classiﬁer that assigns identical clusters to all nearest neighbors and signiﬁcantly improved the clustering performance.
Revised: Based on the observation that K nearest neighbors of each feature point belong to the same ground-truth cluster with a high probability, it trains a classiﬁer that assigns identical cluster id to all nearest neighbors, which signiﬁcantly improved the clustering performance.

Original: Unlike the prior methods where the cluster centroids are not learned in an end-to-end manner or not the subject of interest, C3-GAN jointly learns the cluster centroids with other parts of the model with clear regularizations on their distribution.
Revised: Unlike these prior methods where the cluster centroids are not learnt in an end-to-end manner or not the subject of interest, C3-GAN investigate and try to improve their distribution for achieving better clustering performance.

Original: Unsupervised Scene Decomposition.
Revised: Unsupervised Scene Decomposition.

Original: Heavy cost of annotating segmentation masks have drew active research efforts on developing unsupervised  methods.
Revised: Heavy cost of annotating segmentation masks have drawn active research efforts on developing unsupervised segmentation methods.

Original: Some works address the task in the information-theoretic perspective (Ji et al., 2019; Savarese et al., 2021), but a majority of the works are based on GAN.
Revised: Some works address the task in the information-theoretic perspective (Ji et al., 2019; Savarese et al., 2021), but a majority of the works are based on GANs.

Original: ReDO (Chen et al.
Revised: ReDO (Chen et al.

Original: (2019)) learns to discern object masks by redrawing a scene, and PerturbGAN (Bielski & Favaro (2019)), that has  separate background and foreground generators, triggers scene decompositionlearning by randomly perturbing foreground images.
Revised: (2019)) learns to infer object masks by redrawing an input scene, and PerturbGAN (Bielski & Favaro (2019)), that has a separate background and foreground generator, triggers scene decomposition by randomly perturbing foreground images.

Original: Meanwhile, a line of works utilizes pre-trained high quality  models such as StyleGAN (Karras et al., 2019; 2020) and
Revised: Meanwhile, another line of works utilizes pre-trained high quality generative models such as StyleGAN (Karras et al., 2019; 2020) and

Original: BigGAN (Brock et al., 2019).
Revised: BigGAN (Brock et al., 2019).

Original: Labels4Free (Abdal et al., 2021) trains a segmentation network on top of the pre-trained StyleGAN utilizing the fact that inputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis.
Revised: Labels4Free (Abdal et al., 2021) trains a segmentation network on top of the pre-trained StyleGAN, utilizing the fact that inputs of each layer of StyleGAN have different degrees of contribution to foreground synthesis.

Original: Voynov & Babenko (2020) and Melas-Kyriazi et al.
Revised: Voynov & Babenko (2020) and Melas-Kyriazi et al.

Original: (2021) explore the latent space of pre-trained GANs to ﬁnd the perturbing directions that can be used to induce segmentation masks.
Revised: (2021) explore the latent space of pre-trained GANs to ﬁnd the perturbing directions that can be used for inducing foreground masks.

Original: While all these methods aim to infer foreground masks by training a mask predictor witha synthesized dataset, or projecting real images into the latent space of the pre-trained GAN, C3-GAN is more focusing on the semantic representations learning ofthe foreground object.
Revised: While all these methods aim to infer foreground masks by training an additional mask predictor with synthesized data, or projecting real images into the latent space of the pre-trained GAN, C3-GAN is rather focusing on the semantic representations learning of foreground objects.

Original: Fine-grained Feature Learning.
Revised: Fine-grained Feature Learning.

Original: For hierarchical image generation, some works (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020) extend InfoGAN with multiple pairs of adversarial networks, and integrate the scene decomposition learning method into the framework.
Revised: For ﬁne-grained feature representations learning, some works (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020) have extended InfoGAN by integrating the scene decomposition learning method into the framework with multiple pairs of adversarial networks.

Original: FineGAN (Singh et al., 2019) ﬁnds multiple latent codes using information theory, and each code is used for sequential generation of a background, a super-class object, and a sub-class object image.
Revised: FineGAN (Singh et al., 2019) learns multiple latent codes to use them for sequential generation of a background, a super-class object, and a sub-class object image, respectively.

Original: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020) extend FineGAN with multiple encoders to directly infer these latent codes from real  images.
Revised: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020) extend FineGAN with multiple encoders to directly infer these latent codes from real images and use them for manipulating images.

Original: Since this autoencoder-based structure could evoke  degenerate solutions where only one generator is responsible for synthesizing the entire image, MixNMatch conducts adversarial learning on the joint distribution of the latent code and image (Donahue et al., 2017), and OneGAN trains the model in two stages, learning the generators ﬁrst andthe encoders next.
Revised: Since this autoencoder-based structure could evoke a degenerate solution where only one generator is responsible for synthesizing the entire image, MixNMatch conducts adversarial learning on the joint distribution of the latent code and image (Donahue et al., 2017), while OneGAN trains a model in two stages by training generators ﬁrst and encoders next.

Original: All of these works  succeeded in generating images in a hierarchical manner, and further suggest that the learned latent codes  can be used for clustering real images into their ﬁne-grained classes. However, these works cannot be applied to datasets unlabeled with bounding box annotations, and require a training of additional networks (classiﬁers).
Revised: Even though these works have succeeded in generating images in a hierarchical manner and learnt latent codes that can be used for clustering real images corresponding to their ﬁne-grained classes, they cannot be applied to datasets that have no object bounding box annotations. Moreover, they require a training of additional classiﬁers using a set of generated images annotated with their latent values.

Original: The proposed model in this paper, C3-GAN, can learn the latent code of foreground region in a completely unsupervised way, and  utilize the discriminator for inferring the clusters of the given dataset.
Revised: The proposed model in this paper, C3-GAN, can learn the latent code of foreground region in a completely unsupervised way, and simply utilize the discriminator for inferring the clusters of a given dataset.

Original:  M ETHOD
Revised:  M ETHOD

Original: Given a dataset X = { x i } Ni =1{+N − 1 i =
+} consisting of single object images, we aim to distribute the set X into Y semantically ﬁne-grained clusters in an unsupervised way.
Revised: Given a dataset X = { x i } {+N − 1 i =
+} consisting of single object images, we aim to distribute data into Y semantically ﬁne-grained classes.

Original: Our method, C3-GAN, is built from the GAN, and the clusters are found in the  feature spaceof the discriminator D , H ∈ R d h  .
Revised: Our GAN-based model infers the clusters of data in the semantic feature space H ∈ R d h of the discriminator D .

Original: The cluster index for each image x i is inferred from the cosine distance betweenembedded image feature h i ∈ R d h and the set of cluster centroids l ∈ R Y × d h , which are the latent codes c mapped onto the space. Speciﬁcally, the index of the closest centroid will be assigned to the query image  xi .
Revised: The feature space H is learnt by maximizing the mutual information between the latent code, which is the input of a generator, and its image observation ˆ x .

Original: For the robustﬁne-grained feature  learning, C3-GAN decomposes the scene into the background and foreground part, and associate the latent code only with the foreground region.
Revised: For more robust feature representations learning, we decompose a scene into a background and foreground region and associate the latent code mainly with the foreground region.

Original: Moreover, it induces the discriminator D to learn the embedding space where each cluster is uniformly distributed with distinctive boundaries, by formulating the information regularization based on the contrastive loss.
Revised: We especially reformulate the information-theoretic regularization to optimize the contrastive loss deﬁned for latent-image feature pairs to induce each cluster to be linearly separated in the feature space.

Original: 𝒄 𝒛 𝑮 𝒇𝒈 𝑮 𝒃𝒈 !
Revised: 𝒄 𝒛 𝑮 𝒇𝒈 𝑮 𝒃𝒈 !

Original: 𝑥 !
Revised: 𝑥 !

Original: 𝑥 !" !
Revised: 𝑥 !" !

Original: 𝑥 #" ⨂ 𝒯 $ RandomMatrix !
Revised: 𝑥 #" ⨂ 𝒯 $ RandomMatrix !

Original: 𝑥 #" ′ ⨁ 𝑫 𝒃𝒂𝒔𝒆 𝑵(𝟎,𝑰) 𝓤(𝟎,𝒀 - 𝟏)
Revised: 𝑥 #" ′ ⨁ 𝑫 𝒃𝒂𝒔𝒆 𝑵(𝟎,𝑰) 𝓤(𝟎,𝒀 - 𝟏)

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER 1 CLUSTER CENTROID FEATURE ( 𝒍 ) IMAGE FEATURE ( 𝒉 )
Revised: CLUSTER 1 CLUSTER CENTROID FEATURE ( 𝒍 ) IMAGE FEATURE ( 𝒉 )

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER
Revised: CLUSTER

Original: CLUSTER 1 𝝍 𝒄 𝝍 𝒙𝒉 𝝍 𝒙𝒓 𝝍 𝒙𝒉 𝑫 𝒃𝒂𝒔𝒆
Revised: CLUSTER 1 𝝍 𝒄 𝝍 𝒙𝒉 𝝍 𝒙𝒓 𝝍 𝒙𝒉 𝑫 𝒃𝒂𝒔𝒆

Original: REALorFAKE?
Revised: REALorFAKE?

Original: ( 𝒓 )
Revised: ( 𝒓 )

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: The proposed C3-GAN is built from FineGAN (Singh et al., 2019), which is in turn based on InfoGAN (Chen et al., 2016).
Revised: Our proposed method is built on FineGAN (Singh et al., 2019), which is in turn based on InfoGAN (Chen et al., 2016).

Original: InfoGAN learns to associate a random latent code c with its observation ˆ x by optimizing the model to maximize the mutual information of the two, I( c, ˆ x ) .
Revised: InfoGAN learns to associate the latent code c with its observation ˆ x by inducing the model to maximize the mutual information of the two, I( c, ˆ x ) .

Original: The latent code c can take various forms depending on the prior knowledge of the factor  to be inferred, and be deﬁned as the uniform categorical distribution when the purpose is to make the categorical inference of the data.
Revised: The latent code c can take various form depending on the prior knowledge of the factor that we want to infer, and is set to follow the uniform categorical distribution when our purpose is to make the categorical inference on given dataset.

Original: FineGAN learns three such latent codes for hierarchical image generation, and each code is  used fora background,a super-class object, anda sub-class object image synthesis respectively.
Revised: FineGAN learns three such latent codes for hierarchical image generation, and each code is respectively used for background, super-class object, and sub-class object image synthesis.

Original: To facilitate this decomposition, FineGAN employs multiple pairs of generator and discriminator and trains an auxiliary background classiﬁer using object bounding box labels.
Revised: To facilitate this decomposition, FineGAN employs multiple pairs of generator and discriminator and trains an auxiliary background classiﬁer using object bounding box annotations.

Original: It is demonstrated that the latent codes for the foreground image generation canalso be  utilized for clustering images according to their ﬁne-grained features.
Revised: It further demonstrates that the latent codes for object image synthesis can be also utilized for clustering an image dataset according to their ﬁne-grained classes.

Original: The inference is made by training additional classiﬁers with synthesized images that are pseudo-labeled with their corresponding latent code values.

Original: Our method differs from FineGAN in that, i) it employs the discriminator D for the clustering without requiringan additional training, and ii) it learns only one latent code c that represents the ﬁne-grained class of an foreground object.
Revised: Our method differs from FineGAN in that, i) it employs the discriminator D to infer clusters without requiring additional training of classiﬁers, and ii) it learns only one latent code c that corresponds to the ﬁne-grained class of a foreground object.

Original: Speciﬁcally, our generator G synthesizes background image ˆ x bg ∈ R 3 × H × W , foreground mask ˆ m ∈ R 1 × H × W and foreground appearance ˆ t ∈ R 3 × H × W with two independent randomvariables, the noisevalue z and the latent code c .
Revised: The separate random noise z is kept, that is another input of the generator, to model variations occurring in a background region.

Original: The noise z ∈ R d
Revised: The noise value

Original: z  is sampled from the normal distribution N (0 , I ) . The latent code c ∈ R Y is represented as 1-hot encoding where the index ksuch that  c k = 1 is sampled from the uniform categorical distribution U (0 , Y − 1) .
Revised: z ∈ R d z is sampled from the normal distribution N (0 , I ) , and the latent code c ∈ R Y is an 1-hot vector where the index k that makes c k = 1 is sampled from the uniform categorical distribution, U (0 , Y − 1) .

Original: The noise value z is used for generating all image components making the layout of the scene, while the latent code c is used only for the foreground synthesis.
Revised: Speciﬁcally, the background generator G bg synthesizes a background image ˆ x bg ∈ R 3 × H × W solely from the random noise z , and the foreground generator G fg synthesizes a foreground mask ˆ m ∈ R 1 × H × W and an object image ˆ t ∈ R 3 × H × W using both z and c .

Original: To add some randomness, we augmented the 1-hot latent code c to alatent variable c (cid:48)
Revised: To model foreground variations, we convert an 1-hot latent code c to a variable c (cid:48)

Original: ∈
Revised: ∈

Original: R d c which is sampled from a gaussian distribution N ( µ c , σ c ) wherevalues of the mean µ c andthe diagonal covariance matrix σ c are conditioned on the original code c (Zhang et al., 2017).
Revised: R d c whose value is sampled from the gaussian distribution N ( µ c , σ c ) where the mean µ c and diagonal covariance matrix σ c are computed according to the original code c (Zhang et al., 2017).

Original: The ﬁnal image ˆ x ∈ R 3 × H × W is composited byapplying the hadamard product to the generated components, as described in Fig.
Revised: The ﬁnal image ˆ x ∈ R 3 × H × W is the composition of generated image components summed by the hadamard product, as described in Fig.

Original: To achieve  fully unsupervised scene decomposition by removing the dependence on the bounding box annotations, we leverage the random afﬁne transformation method inspired by PerturbGAN (Bielski & Favaro, 2019).
Revised: To achieve a fully unsupervised method, we leverage the scene decomposition method proposed by PerturbGAN (Bielski & Favaro, 2019).

Original: In speciﬁc, scene decomposition is triggered by perturbing foreground components ˆ m and ˆ t with the  afﬁne transformation matrix T θ right before the ﬁnal image composition.
Revised: In speciﬁc, scene decomposition is triggered by perturbing foreground components ˆ m and ˆ t with the random afﬁne transformation matrix T θ right before the ﬁnal image composition.

Original: The parameters θ of the transformation matrix T θ include  rotation angle, scaling factor and translation distance, and they are all randomly sampled from the uniform distributions with predeﬁned value ranges.
Revised: The parameters θ of the random matrix T θ include a rotation angle, scaling factor and translation distance, and they are all randomly sampled from the uniform distributions with predeﬁned value ranges.

Original: We also employ the regularization functions that enforce foreground mask ˆ m to be more like a hard mask and to take up the reasonable proportions of the scene,

Original: [Equation] In summation, the  generation process is described as:
Revised: In summation, the image generation process is described as below:

Original: [Equation]
Revised: [Equation]

Original: 3. C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING
Revised: 3. C ONTRASTIVE F INE - GRAINED C LASS C LUSTERING

Original: We assume that the dataset X would be effectively clustered when i) it explicitlyforms discernable cluster boundaries in the embedding space H  and ii) each cluster centroid l  ∈ R d h exhibits distinct semantic features.
Revised: We assume that data would be well clustered when i) they form explicitly discernable cluster boundaries in the embedding space H , and ii) each cluster centroid l y ∈ R d h condenses distinct semantic features.

Original: This isexactly the  space that the discriminator of InfoGAN aims to approach when the latent code c is sampled from the uniform categorical distribution U (0 , Y − 1) (Chen et al., 2016).
Revised: This is the exact space that the discriminator of InfoGAN aims to approach when the latent code c is sampled from the uniform categorical distribution, U (0 , Y − 1) .

Original: However, while jointly optimizing an adversarial loss, the model has a possibility of falling into the mode collapse  where a generator G covers only  few conﬁdent modes (classes) to easily deceive the discriminator D .
Revised: However, since it jointly optimizes the adversarial loss, the model has a possibility of falling into the mode collapse phenomenon where the generator G covers only a few conﬁdent modes (classes) to easily deceive the discriminator D .

Original: This is  why InfoGAN cannot infer the classes on real images.
Revised: This is the reason why InfoGAN lacks the ability of inferring clusters on real image datasets.

Original: To handle this problem, we propose a new form of the auxiliary probability Q ( c |  x ) that represents the mutual information between the latent code c and image  x .
Revised: To handle this problem, we propose a new form of an auxiliary probability Q ( c | ˆ x ) that represents the mutual information between the latent code c and image observation ˆ x .

Original: Let us  describe the  mathematical deﬁnition in details.
Revised: Let us now describe the objective functions with mathematical deﬁnitions.

Original: The discriminator D aims to learn  adversarial features r ∈ R for  image authenticity discrimination, and image features h ∈ R d h for optimizing the information-theoretic regularization.
Revised: The discriminator D aims to learn an adversarial feature r ∈ R for the image authenticity discrimination, and a semantic feature h ∈ R d h for optimizing the information-theoretic regularization.

Original: The features are encoded from separate branches ψ rx and ψ hx , which has split at the end of the base encoder of the discriminator D base .
Revised: The features are encoded from separate branches ψ rx and ψ hx , which were split at the end of the base encoder of the discriminator D base .

Original: The adversarial features r are learned with the hinge loss as shown in the equation below:
Revised: The adversarial feature r is learnt with the hinge loss as represented in the equations below:

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, the image features h can be considered as logits in a classiﬁcation model.
Revised: Meanwhile, the semantic feature h can be considered as a logit in a classiﬁcation model.

Original: In a supervisedlearning setting wherethe ground-truth class labels are given,the data mapped into the embedding space gradually forms the cluster boundaries as the training proceeds, like the one in Fig. 1 (b).
Revised: In a supervised setting where ground-truth class labels are given, data mapped into an embedding space gradually form discernable cluster boundaries as the training proceeds, similar to the one in Fig.(b).

Original: However,  it is difﬁcult to learn such spacein the given problem setting simply by maximizing the mutual information between the latent code c and its observation ˆ x, I( c ; ˆ x ) , for the aforementioned reasons.
Revised: However, in an unsupervised setting, it is difﬁcult to attain such space simply by maximizing the mutual information between the latent code c and its observation ˆ x for the aforementioned reasons.

Original: Our solution is to formulate the auxiliary probability Q ( c |  x ) based on the  contrastive loss.
Revised: Our solution is to formulate an auxiliary probability Q ( c | ˆ x ) in the form of contrastive loss.

Original: Speciﬁcally, we map the latent code c onto the embedding space of the discriminator with  simple linear layer ψ c , and let it act as a cluster centroid  l  that pulls the feature of the image h  that has generated with this speciﬁc value.
Revised: We map the latent code c onto the embedding space H with a simple linear layer ψ c , and let it act as a cluster centroid ( l ) that pulls semantic features of images ( h ) that were generated from its speciﬁc value.

Original: By setting the relation with other latent codes as negative pairs,the cluster centroids would be pushing each others so that they are distributed with distinct boundaries. The equation of the proposed formulation is as below:
Revised: By setting the relations with other latent codes as negative pairs, cluster centroids are set to be pushing each others and form distinguishable boundaries in the embedding space.

Original: [Equation] where sim( a, b ) is the cosine distance between vectors a and b , y is the index of the cluster, and τ is a temperature of the softmax function. I Y denotes the identity matrix with size Y . With this function, we expectthat the image feature h would be mappedclosest to  k -th cluster, where k is the index of its corresponding latent code csuch that  c k = 1 , while distances to other centroids l y would be far enough so that the value of q k is close to 1.
Revised: Speciﬁcally, we expect the semantic feature h to be mapped to the k -th cluster centroid, where k is the index of its corresponding latent code c that makes c k = 1 , while distances to other centroids l y are far enough to maximize the mutual information of the two.

Original: We additionally maximize the mutual information between the latent code c andthe masked foreground images ˆ x fg to further assist the scene decomposition learning.
Revised: To enhance the robustness of the model and assist the scene decomposition learning, we additionally maximize the mutual information between the latent code c and masked foreground images ˆ x fg .

Original: So the  information-theoretic objectives are deﬁned as:
Revised: To sum up, the proposed information-theoretic objectives are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Revised: where sim( a, b ) is the cosine distance between vectors a and b , y is an index of a cluster, and τ is the temperature of the softmax function.

Revised: I Y denotes the identity matrix with the size of Y × Y .

Original:  Following the prior works (Ji et al., 2019; Van Gansbeke et al., 2020), we adopted overclustering strategy so that the modelcan learn more expressive features.
Revised: Additional Regularizations Following the prior works (Ji et al., 2019; Van Gansbeke et al., 2020), we adopt the overclustering strategy to help the model learn more expressive feature set.

Original: We also regularize the prediction on  real images p ∈ R Y to further reduce the gap between real and fake images. Speciﬁcally, we minimize the entropyof p , H ( p ), to promote each data to be mapped to  one cluster  with high conﬁdence and the KL divergence between the averaged prediction ¯p and the uniform distribution u to avoid a degenerated solution where  few clusters are overly allocated.
Revised: We also regularize the prediction on a real image p ∈ R Y by minimizing its entropy H ( p ) to promote each data to be mapped to only one cluster id with high conﬁdence, along with the minimization of the KL divergence between batch-wisely averaged prediction ¯p and the uniform distribution u , that is for avoiding a degenerated solution where only a few clusters are overly allocated.

Revised: Furthermore, we optimize the contrastive loss (Chen et al., 2020) deﬁned for real image features h to assist the semantic feature learning.

Original: The regularizations  are as below:
Revised: To summarize, the regularizations on real images are as follows: [Equation] [Equation]

Revised: We also enforce a foreground mask ˆ m to be more like a hard mask and take up a reasonable portion of a scene by employing below regularization function.

Original: [Equation]
Revised: [Equation]

Original: In summary, our C3-GAN is trained  optimizing the following objective functions iteratively:
Revised: In sum, C3-GAN is trained alternately optimizing the following objective functions for D and G :

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETTING
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETTING

Original: Datasets.
Revised: Datasets.

Original: We tested our method on 4 datasets which contain single object images.
Revised: We tested our method on 4 datasets that consist of single object images.

Original: i) CUB
Revised: i) CUB

Original: (Wah et al., 2011): 5,994 training and 5,794 test images of 200 bird species.
Revised: (Wah et al., 2011): 5,994 training and 5,794 test images of 200 bird species.

Original: ii) Oxford Flower (Nilsback & Zisserman, 2008): 2,040 training and 6,149 test images of 102 ﬂower categories.

Original: iii)
Revised: ii)

Original: Stanford Cars (Krause et al., 2013): 8,144 training and 8,041 test images of 196 car models.
Revised: Stanford Cars (Krause et al., 2013): 8,144 training and 8,041 test images of 196 car models.

Original: iv)
Revised: iii)

Original: Stanford Dogs (Khosla et al., 2011): 12,000 training and 8,580 test images of 120 dog species.
Revised: Stanford Dogs (Khosla et al., 2011): 12,000 training and 8,580 test images of 120 dog species.

Original: Since the
Revised: iv) Oxford Flower (Nilsback & Zisserman, 2008): 2,040 training and 6,149 test images of 102 ﬂower categories.

Original: CUB and Oxford Flower datasets have a small number of training images, all the experiments were conducted using the entire data as in prior works  (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020).
Revised: Due to the small number of training images, all models for CUB and Oxford Flower datasets were trained with the entire dataset as the prior works did (Singh et al., 2019; Li et al., 2020; Benny & Wolf, 2020).

Original: Implementation Details.
Revised: Implementation Details.

Original: The weights of  loss terms ( λ 0 , λ 1 , λ 2 , λ 3  ) are set as (0.1, 1, 5, 1), and the temperature τ is set as 0.1.
Revised: The weights of the loss terms ( λ 0 , λ 1 , λ 2 , λ 3 , λ 4 ) are set as (5, 1, 1, 0.1, 1), and the temperature τ is set as 0.1.

Original: The models are trained with Adam optimizer with learning rate of 0.
Revised: We utilized Adam optimizer of which learning rate is 0.

Original: and  momentum coefﬁcients of (0.5, 0.999).
Revised: and values of momentum coefﬁcients are (0.5, 0.999).

Original: We described the network speciﬁcation in Appendix A.1, and the value ranges ofparameters for the random afﬁne transformation and the number of clusters Y  in Appendix A.2.
Revised: The architectural speciﬁcation of C3-GAN and the values of hyperparameters, such as the parameter ranges of the random afﬁne transformation and the number of clusters Y can be found in Appendix A.1 and Appendix A.2, respectively.

Original: Baselines.
Revised: Baselines.

Original: We ﬁrst compare C3-GAN to IIC (Ji et al., 2019) in order to emphasize the difference between the coarse-grained and ﬁne-grained class clustering tasks since it achieved decent clustering results with sobel ﬁltered images where color and texture information is discarded.
Revised: We ﬁrst compared C3-GAN with IIC (Ji et al., 2019) in order to emphasize the difference between the coarse-grained and ﬁne-grained class clustering task, since it achieved decent performance with sobel ﬁltered images where color and texture informations are discarded.

Original: To claim that optimizingthe instance discrimination task is not enough for ﬁne-grained feature representations learning, we also compare with the simple baseline of SimCLR+K-Means (Chen et al., 2020).
Revised: We also experimented with simple baseline of SimCLR+K-Means (Chen et al., 2020) to claim that optimizing instance discrimination task is not enough for ﬁne-grained feature learning.

Original: The work that shows remarkable result on the coarse-grained class clustering, SCAN (Van Gansbeke et al., 2020), are also included to evaluate their method that induces consistent predictions between nearest neighbors on the ﬁne-grained class clustering task.
Revised: SCAN (Van Gansbeke et al., 2020), the work that shows a remarkable coarse-grained class clustering performance, is also compared to investigate whether the method of inducing consistent predictions for all nearest neighbors would be helpful for the ﬁne-grained class clustering task as well.

Original: In addition to InfoGAN (Chen et al., 2016), the reported results of FineGAN (Singh et al., 2019),
Revised: Finally, we compared with the methods that learn hierarchically ﬁne-grained feature representations, such as FineGAN (Singh et al., 2019),

Original: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020) were compared to claim the efﬁcacy of our proposed formulation of the auxiliary probability based on the contrastive loss.
Revised: MixNMatch (Li et al., 2020) and OneGAN (Benny & Wolf, 2020), including their base model InfoGAN (Chen et al., 2016), to claim the efﬁcacy of the proposed formulation that is based on the contrastive loss.

Original: For the fair comparison, we mainly compared with theresults of the unsupervised version of FineGAN and MixNMatch, which are trained by pseudo-labeling the edges of real images as background regions.
Revised: We mainly compared with the unsupervised version of them that is trained with pseudo-labeled bounding boxes which assign edges of real images as background region.

Original: 4. F INE - GRAINED C LASS C LUSTERING R ESULTS
Revised: 4. F INE - GRAINED C LASS C LUSTERING R ESULTS

Original: Quantitative results.
Revised: Quantitative results.

Original: We evaluate the clustering performance with two metrics: Accuracy (Acc) and Normalized Mutual Information (NMI).
Revised: We evaluated clustering performance with two metrics: Accuracy (Acc) and Normalized Mutual Information (NMI).

Original: The  accuracy is calculated based on the optimal mapping betweenthe cluster indices andthe real classes inferred from the Hungarian algorithm (Kuhn & Yaw, 1955).
Revised: The score of accuracy is calculated following the optimal mapping between cluster indices and real classes inferred by Hungarian algorithm (Kuhn & Yaw, 1955) for a given contingency table.

Original: The results are presented  in Table 1.
Revised: We presented the results in Table 1.

Original: As shown in Table 1, our C3-GAN outperforms other methods with remarkable margins in terms of Acc on all datasets.
Revised: As it can be seen, C3-GAN outperforms other methods by remarkable margins in terms of Acc on all datasets.

Original: For NMI, our method presents better or comparable performances as well.
Revised: Our method presents better or comparable performance in terms of NMI scroes as well.

Original: Thelowest results of IIC underlines that understanding only structural characteristic is not enough for the ﬁne-grained class clustering task.
Revised: The results of IIC underline that understanding only structural characteristic is not enough for the ﬁne-grained class clustering task.

Revised: From the results of SCAN (2nd rank) and MixNMatch (3rd rank), we can conjecture that both requiring local disentanglement of the embedding space and extracting foreground features via scene decomposition learning can be helpful for improving the ﬁne-grained class clustering performance.

Original: It is difﬁcult to decide which of the GAN-based methods or the SSL-based methods are better, since both approaches have similar range of scores on every metric and display globally entangled feature spaces as shown in Fig.
Revised: However, it is difﬁcult to determine whether the GAN-based methods or the SSLbased methods are better because both approaches have similar score ranges for all metrics and display globally entangled feature spaces as shown in Fig.

Original: 3 (a) and (b).
Revised: 3 (a) and (b).

Original: From the results of SCAN (2nd rank) and MixNMatch (3rd rank), we conjecture that circumventing the problem by requiring local disentanglement of the embedding space and extracting foreground features via scene decomposition are helpful for improving the ﬁne-grained [Table] FineGAN
Revised: Since C3-GAN resembles

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table]
Revised: MixNMatch to some extent, its state-of-the-art scores can be partially explained by the fact that it learns decomposed foreground features, but the embedding space of C3-GAN visualized in Fig.

Original: [Figure] clustering performance. The state-of-the-art scores of C3-GAN are not only beneﬁted from the scene decomposition learning, but also boosted from the proposed formulation of Q ( c |  x ) that enforcesthe clusters to be distinctively distributed in the embedding space as visualized in Fig. 3 (c).
Revised: 3 (c) implies that the performance of it has been further improved from our formulation of Q ( c | ˆ x ) that enforces clusters to be distinctively distributed in the embedding space.

Original: Furthermore, we present multiple qualitative results  on ﬁne-grained clustering tasks in the  Appendix A.3, which show the superiority of our method compared to SCAN and MixNMatch.
Revised: We additionally presented the results of qualitative analysis on clustered images inferred by the top three methods in Appendix A.3, which demonstrate the excellence of our method compared to prior works.

Original: Ablation Study.
Revised: Ablation Study.

Original: We performed ablation study regarding three factors: i)
Revised: We conducted an ablation study regarding three factors: i)

Original: Overclustering, ii)
Revised: Overclustering, ii)

Original: Perturbing foreground images via random afﬁne transformation, and iii) Our formulation of the auxiliary probability Q ( c | x ) .
Revised: Foreground perturbation, and iii) The information-theoretic regularization based on the contrastive loss.

Original: We investigate the clustering performance  on CUB dataset by removing each factor and the results are shown in Table 2.
Revised: We investigated clustering performance of C3-GAN on CUB dataset by removing each factor. The results can be seen in Table 2.

Original: The largest performance degradation is induced by changing the formulation of the auxiliary probability Q ( c | x ) to a trivial softmax function.
Revised: The largest performance degradation was caused by changing the formulation of the information-theoretic regularization to a trivial softmax function.

Original: This result implies that the high  degree of cluster separation, which achieved with our proposed method, is strongly correlated with the clustering performance (Wang & Isola, 2020).
Revised: This result implies that clustering performance has a high correlation with the degree of separability of clusters, which was signiﬁcantly improved by our method.

Original: We observe that C3-GAN fails to decompose background and foreground parts when the random perturbation T θ is not applied.
Revised: Regarding the result of ii), we observed that C3-GAN fails to decompose a scene when the foreground perturbation was not implemented.

Original: The signiﬁcant performance degradation suggests that the method of extracting onlythe foreground features has a non-negligible amount of beneﬁt inthe clustering performance.
Revised: This degenerated result suggests that the method of extracting only foreground features has a non-negligible amount of inﬂuence in clustering performance.

Original: Overclustering is also helpful for boostingthe performance by allowing a larger capacity for the feature representations learning.
Revised: Lastly, the result of i) implies that the overclustering strategy is also helpful for boosting performance since it allows a larger capacity for model to learn more expressive feature set.

Original: We added more ablation results on overclustering inthe Appendix A.2.[Table] i
Revised: The additional analysis on hyperparameters can be found in Appendix A.2.

Original: [Table]
Revised: [Table]

Original:  – Overclustering
Revised: i) – Overclustering

Original: [Table]
Revised: [Table]

Original: ii)
Revised: ii)

Original: – Perturbing foreground images via random afﬁne transformation
Revised: – Foreground perturbation

Original: [Table]
Revised: [Table]

Original: iii
Revised: iii

Original: [Table]
Revised: [Table]

Original: – Contrastive loss based [Table] [Table] [Figure]
Revised: theoretic

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We present (a) the synthesized images with the predicted cluster indices from real images and (b) the randomly synthesized images with sampled input variables z and c

Original: 4. I MAGE G ENERATION R ESULTS 
Revised: 4. I MAGE G ENERATION R ESULTS Qualitative Results

Original: Qualitative Results.

Original: Fig. 4 presents images synthesized by C3-GAN trained on CUB dataset.
Revised: For qualitative evaluation, we analyzed the results on CUB dataset.

Original: First, we plot the synthesized novel scenes with thepredicted cluster indices of real images in Fig. 4(a).
Revised: In Fig.(a), we displayed synthesized images along with their decomposed scene elements. The images were generated with the cluster indices of real images that were predicted by the discriminator.

Original: Considering the identical characteristics ofsynthesized foreground objects, C3-GAN achieved to learn cluster centroids with distinctive ﬁne-grained features set so that it can also clearly mapped with real images.
Revised: Considering the consistency of foreground object features within the same class and the uniqueness of each class feature, we can assume that C3-GAN succeeded in learning the set of clusters in which each cluster represents its unique characteristic explicitly.

Original: To more investigate the characteristics of  two input variables, c and z , we plot theresults synthesized  by ﬁxing one of the two while diversifying the other one in Fig. 4(b).
Revised: Further, to investigate the roles of the two input variables, c and z , we analyzed the change in the synthesized images when they were generated by ﬁxing one of the two and diversifying the other, as shown in Fig. 4 (b).

Original: When c is ﬁxed, allthe generated images contain the same bird species while the pose of the bird and backgrounds vary due to the variation of z value.
Revised: As it can be seen, if c is ﬁxed, all images depict the same bird species, but the pose of a bird and background vary depending on the value of z .

Original: On the contrary, for the  variable c and ﬁxed z setting, we could observe images of various bird species with identical pose and background features.
Revised: Conversely, when generated under the condition of variable c and ﬁxed z , images of various bird species with the same pose and background were observed.

Original: We also plot the results of other datasets in the appendix A.4.
Revised: The results on other datasets show the same trend. Please refer to Appendix A.4.

Original: We observe that C3-GAN also effectively alleviates the mode collapse issue from Fig.
Revised: It is also notable that C3-GAN can effectively alleviate the mode collapse issue with the proposed method.

Original: The images are synthesized with the cluster index corresponding to the class of real image.
Revised: Fig.

Original: While C3-GAN presents intra -class color, shape, and layoutvariation for a givenﬂower class, the results of other baselines differ only in layout.
Revised: 5 present that only C3-GAN reﬂects intra-class deviations of various factors such as color, shape, and layout for the given class, while other baselines produce images only with the layout variation.

Original: We conjecture that this is because the learnt embedding space of the discriminator  represents the real data distribution so well, that the generatorgets a stronger signal for learning to synthesize realistic images in addition to the adversarial loss. Through the qualitative
Revised: We conjecture that this performance was achieved because the learnt embedding space of the discriminator better represents the real data distribution, allowing the generator to get quality

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table] [Figure]
Revised: Only C3-GAN generates images reﬂecting the intra -class color, shape, and layout variations of the real image set.

Original: [Table]
Revised: [Table]

Original: investigation of synthesized images, wealso found thatC3-GAN was the only one that succeeded in  scene decomposition learning in an unsupervised way.
Revised: signals for the adversarial learning. Additionally, we found that only C3-GAN succeeded in the scene decomposition learning when annotations were not utilized.

Original:  FineGAN and MixNMatch  trained without  bounding box annotations fell intoa degenerate solution where the entire image is synthesized from the background generator.
Revised: In fact, when FineGAN and MixNMatch were trained without object bounding box annotations, they fell into degenerate solutions where the entire image is synthesized from the background generators.

Original: Quantitative Results.
Revised: Quantitative Results.

Original: We evaluate image synthesis performance based on the scores of Frechet Inception Distance (FID) and Inception Score (IS), in addition to the score of reverse KL which measures the distance between the predicted class distribution of real  and synthesized images.
Revised: We also quantitatively evaluated the image synthesis performance based on the scores of Frechet Inception Distance (FID) and Inception Score (IS). We also considered the scores of reverse KL which measures the distance between the predicted class distribution of real images and synthesized images.

Original: We compared C3-GAN withthe GAN-based models  trained in a fully unsupervised setting.
Revised: We compared C3-GAN with GAN-based models that were trained without object bounding box annotations.

Original: The scores of IS and reverse KL were measured with Inception networks  ﬁne-tuned on eachﬁne-grained dataset using the method of Cui et al.
Revised: The scores of IS and reverse KL were measured with Inception networks that were ﬁne-tuned on each dataset using the method of Cui et al.

Original: (2018) 1 , and the results are displayed in Table 3.
Revised: The results are displayed in Table 3.

Original: C3-GAN presents  state-of-the-art or comparable performance in terms of FID and Reverse KL.
Revised: C3-GAN presents the state-of-the-art or comparable performance in terms of all metrics.

Original: With the qualitative results together, these results imply our C3-GAN has competitive capability for ﬁne-grained object image generation.
Revised: This result means that, in addition to the class clustering performance, C3-GAN has competitive performance in generating ﬁne-grained object images.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this study, we proposed a new GAN-based method, C3-GAN, for unsupervised ﬁne-grained class clustering which is more challenging and less explored than  coarse-grained clustering tasks.
Revised: In this study, we proposed a new GAN-based method, C3-GAN, for unsupervised ﬁne-grained class clustering which is more challenging and less explored than a coarse-grained clustering task.

Original: To improve  ﬁne-grained class clustering performance, we formulate the information-theoretic regularization based on the contrastive loss.
Revised: To improve the ﬁne-grained class clustering performance, we formulate the information-theoretic regularization based on the contrastive loss.

Original: Also, a scene decomposition-based approach is incorporated into the framework to enforce the model to learn representation focusing on the details of foreground images.
Revised: Also, a scene decomposition-based approach is incorporated into the framework to enforce the model to learn features focusing on a foreground object.

Original: We further enhance the performance by adopting the overclustering approach.

Original: Extensive experiments show  our C3-GAN not only outperforms previous ﬁne-grained clustering methods but also synthesizes ﬁne-grained object images with comparable quality while reducing mode collapse that previous state-of-the-art GAN methods suffer from.Since our main goal is unsupervised ﬁnegrained clustering rather than image generation, C3-GAN still has room to be improved despite its comparable results.
Revised: Extensive experiments show that our C3-GAN not only outperforms previous ﬁne-grained clustering methods but also synthesizes ﬁne-grained object images with comparable quality, while alleviating mode collapse that previous state-of-the-art GAN methods have been suffering from.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Remarkable advancement of generative models has played a crucial role as AI tools for text (Brown et al., 2020;
Revised: Remarkable advancement of generative models has played a crucial role as AI tools for text (Brown et al., 2020;

Original: Kim et al., 2021), image (Karras et al., 2020; Choi et al., 2020b), audio (Sisman et al., 2020), and multimodal content generation (Tian et al., 2021; Yan et al., 2021).
Revised: Kim et al., 2021), image (Karras et al., 2020; Choi et al., 2020b), audio (Sisman et al., 2020), and multimodal content generation (Tian et al., 2021; Yan et al., 2021).

Original: However, as its side effect, many malicious applications have been reported as well, thus leading to severe societal problems, such as deepfake (Dolhansky et al., 2020), fake news (Lazer et al., 2018), and generation biased by training data (Gupta et al., 2021).
Revised: However, as its side effect, many malicious applications have been reported as well, thus leading to severe societal problems, such as deepfake (Dolhansky et al., 2020), fake news (Lazer et al., 2018) and generation biased by training data (Gupta et al., 2021).

Original: Our work might be an extension of the harmful effects of generative models.
Revised: Our work might be an extension of the harmful effects of generative models.

Original: On the contrary, generative models can be a solution to alleviate these side effects via data augmentation (Yang et al., 2019; Choi et al., 2020a).
Revised: On the contrary, generative models can be a solution to alleviate these side effects via data augmentation (Yang et al., 2019; Choi et al., 2020a).

Original: In particular, our method can contribute to alleviating data bias and ﬁne-grained class imbalance.
Revised: In particular, our method can contribute to alleviating data bias and ﬁne-grained class imbalance.

Original: Therefore, many endless efforts are required to make these powerful generative models beneﬁt humans.
Revised: Therefore, many endless efforts are required to make these powerful generative models beneﬁt humans.

Original: R EPRODUCIBILITY S TATEMENT
Revised: A CKNOWLEDGMENTS

Original: We described the datasets and baselines compared with our method in detail in 4.1.
Revised: This work was experimented on the NAVER Smart Machine Learning (NSML) platform (Sung et al., 2017; Kim et al., 2018).

Original: The details of our model architecture and hyperparameter setup are also explained in A.1 and A.2. And we plan to open the source code of C3-GAN for reproducibility
Revised: We are especially grateful to Jun-Yan Zhu and NAVER AI Lab researchers for their constructive comments.

Original: A A PPENDIX A. N ETWORK A RCHITECTURE
Revised: A A PPENDIX A. N ETWORK A RCHITECTURE

Original: We presentthe detailed description of  architecture ofthe generator and the discriminator for our C3-GAN in Table 4 and Table 5, respectively.
Revised: We present detailed description of the architecture of C3-GAN in Table 4 and Table 5.

Original: The colored dots ( • , • , • ) indicate that  outputs ofthe earlier layers are used asinputs for the  later layers.
Revised: The colored dots ( • , • , • ) are for indicating that the outputs of earlier layers are used as the inputs of later layers.

Original: The training is done with 2 NVIDIA-V100 GPUs, and we optimized the model until the convergence of the FID score
Revised: The training was done with 2 NVIDIA-V100 GPUs, and we optimized a model until the convergence of the FID score

Original: [Table] [Table]
Revised: [Table] [Table]

Original: • Conv( a , b ) : 2D Convolution layer with kernel size a and stride b • 
Revised: • Conv( a , b ) : 2D Convolution layer with kernel size a and stride b • GLU : Gated Linear Units

Original: A. A DDITIONAL A BLATION S TUDY Random afﬁne transformation.
Revised: A. A DDITIONAL A BLATION S TUDY Random afﬁne transformation.

Original: To validate the effects of random afﬁne transformation, we experimented with two types of purtubing policies such as weak and strong perturbation.
Revised: We experimented with two types of purtubing policy, a weak and strong random afﬁne transformation.

Original: Their detailed value ranges  are presented in Table 6.
Revised: The detailed value ranges of their parameters are presented in Table 6.

Original: The type of perturbation to use for each dataset was determined by manually checkingthe results of randomly perturbed real images.
Revised: The type of perturbation used for each dataset was determined by manually checking randomly perturbed real images.

Original: The strong perturbation was  used for CUB dataset, and the rest datasets were trained with the weak perturbation.
Revised: The strong perturbation was only used for CUB dataset, and the rest datasets used the weak perturbation.

Original: [Table]
Revised: [Table]

Original: Cluster numbers for each dataset.
Revised: Overclustering.

Original: We use four datasets for  evaluation, including CUB,Oxford Flower, Standford Cars,and Stanford Dogs.
Revised: We use four datasets for the evaluation, including CUB, Standford Cars, Stanford Dogs and Oxford Flower.

Original: As depicted in 3.2, we employ the overclustering approaches to improve the performances, which is to set thecluster number  to be multiple times as the actual number of classes of the dataset.
Revised: As we described in 3.2, we employ the overclustering policy which is to set the number of clusters to be multiple times of the actual number of classes.

Original: For ﬁnding optimal setting, we compared the results of C3-GAN by setting the number of clusters as 1, 2, and 3 times the actual number of classes.
Revised: To ﬁnd the optimal setting, we compared the results of C3-GAN by setting the number of clusters as 1, 2, and 3 times the actual number of classes.

Revised: The cluster numbers for each dataset were determined based on the results in Table 7.

Original: We found that  performance  tends to improve as the number of clusters increases.
Revised: We found that the performance generally tends to improve as the number of clusters increases.

Original: This implies that overclustering is indeed helpful for  expressive feature learning.
Revised: This implies that overclustering is indeed helpful for the expressive feature learning.

Original: The cluster numbers for each dataset were determined based on the results in Table 7.

Original: To further investigate the results when we are not aware of the actual number of classes, we additionally conducted all experiments by setting the number of clusters as 100 (underclustering) or 500 (overclustering).
Revised: To further investigate the results when we are not aware of the actual number of classes, we additionally conducted all experiments by setting the number of clusters as 100 (underclustering) or 500 (overclustering).

Original: The results are represented in Table 8.
Revised: The results are represented in Table 8.

Original: For the underclustering setting, we only report the scores of NMI, since some classes have no matching cluster indices.
Revised: For the underclustering setting, we only report the scores of NMI, since some classes have no matching cluster indices.

Original: Despite the less promising results of the underclustering, this  implies that our method can be applied to  datasets whose cluster size is not available if we set a large enough number of clusters. 
Revised: Despite the less promising performance of the underclustering case, this result implies that our method can be applied to any datasets whose cluster size is not available, if we set a large enough number of clusters. [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. Q UALITATIVE A NALYSIS OF C LUSTERING R ESULTS
Revised: A. Q UALITATIVE A NALYSIS OF C LUSTERING R ESULTS

Original: We present the clustered images by our C3-GAN on two classes for CUB and Oxford Flowers datasets, comparing the results of MixNMatch and SCAN in Figs.
Revised: We present the clustered images of CUB and Oxford Flowers datasets, along with the results of MixNMatch and SCAN in Figs.

Original: 6 and 7.
Revised: 6 and 7.

Original: From Fig. 6, our method presents better clustered results outperforming two comparable methods.
Revised: From these results, we can assume that our method is better at clustering compared to the baseline methods.

Original: It is worth noting that eventhe objects in images incorrectly clustered by C3-GAN look very similar to given object class while thecluster results of othertwo methods containimages of objects that seldom resemble the given class object.
Revised: It is worth noting that even incorrectly assigned images look very similar with the given bird species for C3-GAN, while the results of other methods contain objects that are quite deviated from the condition.

Original: We can conﬁrm the consistent results on the Flower dataset as  shown in Fig.
Revised: We could observe the similar trend for Oxford Flowers dataset, as it is shown in Fig.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. 
Revised: A. A DDITIONAL IMAGE GENERATION RESULTS

Original: A DDITIONAL IMAGE GENERATION RESULTS

Original: A.4.
Revised: A.4.

Original: CUB
Revised: CUB

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background Background
Revised: Background Background

Original: Background + Foreground + Foreground + Foreground
Revised: Background + [Table] + [Table]

Revised: +

Revised: [Table]

Original: Fixed c and variable z
Revised: Fixed c and variable z

Original: Varible c and ﬁxed z
Revised: Varible c and ﬁxed z

Original: A.4.
Revised: A.4.

Original: S TANFORD C ARS
Revised: S TANFORD C ARS

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background
Revised: Background

Original: Background
Revised: Background

Original: Background + Foreground + Foreground + Foreground
Revised: Background + Foreground + Foreground + Foreground

Original: Fixed c and variable z
Revised: [Table]

Original: Varible c andﬁxed z
Revised: and

Original: [Table]
Revised: [Table]

Original: S
Revised: z

Revised: Varible

Original: [Table]
Revised: [Table]

Revised: and ﬁxed z

Revised: A.4. S TANFORD D OGS

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background
Revised: Background

Original: Background
Revised: Background

Original: Background + Foreground + Foreground + Foreground
Revised: Background + Foreground + Foreground + Foreground

Original: Fixed c and variable z
Revised: Fixed c and variable z

Original: Varible c and ﬁxed z
Revised: Varible c and ﬁxed z

Original: [Table] 
Revised: A.4.

Original: O
Revised: O

Original: [Table]
Revised: XFORD

Original: F
Revised: F

Original: [Table]
Revised: LOWER

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Real
Revised: Real

Original: Fake images
Revised: Fake images

Original: Background
Revised: Background

Original: Background
Revised: Background

Original: Background + Foreground + Foreground + Foreground
Revised: Background + Foreground + Foreground + Foreground

Original: Fixed c and variable z Varible c and ﬁxed z
Revised: Fixed c and variable z Varible c and ﬁxed z