Original: COCO-GAN: CO NDITIONAL CO ORDINATE G ENERATIVE A DVERSARIAL N ETWORK
Revised: COCO-GAN: CO NDITIONAL CO ORDINATE G ENERATIVE A DVERSARIAL N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent advancements on Generative Adversarial Network (GAN) have inspired a wide range of works that generate synthetic images.
Revised: Recent advancements on Generative Adversarial Network (GAN) have inspired a wide range of works that generate synthetic images.

Original: However,the current processes have to generate an entire image at once, and therefore resolutions are limited by memory or computational constraints.
Revised: However, current processes have to generate an entire image at once, and therefore resolutions are limited by memory or computational constraints.

Original: In this work, we propose 
Revised: In this work, we propose COnditional

Original: COnditional COordinate GAN (COCO-GAN), which generates a speciﬁc patch of an image conditioned on a spatial position rather than the entire image at a time.
Revised: COordinate GAN (COCO-GAN), which generates a speciﬁc patch of an image conditioned on a spatial position rather than the entire image at a time.

Original: The generated patches are later combined together to form a globally coherent full-image.
Revised: The generated patches are later combined together to form a globally coherent full-image.

Original: With this process, we show that the generated image can achieve competitive quality to state-of-the-arts and the generated patches are locally smooth between consecutive neighbors.
Revised: With this process, we show that the generated image can achieve competitive quality to state-of-the-arts and the generated patches are locally smooth between consecutive neighbors.

Original: One direct implication of the COCO-GAN is that it can be applied onto any coordinate systems including the cylindrical systems which makes it feasible for generating panorama images.
Revised: One direct implication of the COCO-GAN is that it can be applied onto any coordinate systems including the cylindrical systems which makes it feasible for generating panorama images.

Original: The fact that the patch generation process is independent to each other inspires a wide range of new applications: ﬁrstly, “Patch-Inspired Image Generation” enables us to generate the entire image based on a single patch.
Revised: The fact that the patch generation process is independent to each other inspires a wide range of new applications: ﬁrstly, “Patch-Inspired Image Generation” enables us to generate the entire image based on a single patch.

Original: Secondly, “Partial-Scene Generation” allows us to generate images within a customized target region.
Revised: Secondly, “Partial-Scene Generation” allows us to generate images within a customized target region.

Original: Finally, thanks to COCO-GAN’s patch generation and massive parallelism, which enables combining patches for generating a full-image with higher resolution than state-of-thearts.
Revised: Finally, thanks to COCO-GAN’s patch generation and massive parallelism, which enables combining patches for generating a full-image with higher resolution than state-of-the-arts.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Human perception has only partial access to the surrounding environment due to the limited acuity area of fovea, and therefore human learns to recognize or reconstruct the world by moving their eyesight.

Original: Computational vision models mostly assume machines always have full access to the whole scene, which is one of the key bottlenecks of modern models when dealing with large ﬁeldof-view and differently-shaped input data in different tasks. This paper explores the idea of implementing spatial disentanglement into generative adversarial networks (GANs) (Goodfellow et al., 2014).
Revised: This paper explores the idea of enforcing both the generator and the discriminator of generative adversarial networks (GANs) (Goodfellow et al., 2014) to deal with only partial views via conditional coordinating. Via training and inference with partial views only, the minimum memory requirement can be largely reduced. However, as shown in Section 3.3, naive approaches fail to generate high quality images, either having clear seams or totally failing to generate reasonable structures.

Original: We propose and develop a new GAN architecture: COnditional COordinate GAN (COCO-GAN).
Revised: We investigate this problem and propose a new GAN architecture: CO nditional COordinate GAN (COCO-GAN).

Original: Given a latent vector and  spatial positions, the generator of COCO-GAN learns to produce image patches independently.
Revised: Given a latent vector and multiple spatial positions, the generator of COCO-GAN learns to produce image patches independently according to the spatial positions.

Original: On the other hand, the discriminator learns to judge whether adjacent generated patches are structurally sound and visually homogeneous.
Revised: On the other hand, the discriminator learns to judge whether adjacently generated patches are structurally sound and visually homogeneous.

Original: The generated patchesthen can directly be used to compose a complete full-image without further post-processing.
Revised: During the inference phase, the generated patches can directly be used to compose a complete full-image without further post-processing.

Original: Thanks to the adversarial loss provided by the discriminator, the composed full-image is locally smooth and globally convincing.
Revised: Owing to the adversarial loss provided by the discriminator, the composed full-image is locally smooth and globally convincing.

Original: We show several randomly-selected full-images generated by COCO-GAN in Figure 2a.
Revised: We show several randomly-selected full-images generated by COCO-GAN in Figure 2a.

Original: In Section 3.2, we visualize the learned spatial disentanglement via the task of spatial positions interpolation.
Revised: In Section 3.2, we visualize the interpolation between two spatial positions, showing COCO-GAN has inter-class continuity as common conditional GANs do.

Original: Further quantitative evaluations with “Frchet Inception Distance” (FID) (Heusel et al., 2017) score are presented in Table 1.
Revised: Further quantitative evaluations with “Frchet Inception Distance” (FID) (Heusel et al., 2017) score are presented in Table 1.

Original: Without additional hyper-parameter tuning, the evaluations on CelebA (Liu et al., 2015) and LSUN (Yu et al., 2015) datasets suggest that COCO-GAN is competitive with other state-of-the-art GANs.
Revised: Without additional hyper-parameter tuning, the evaluations on CelebA (Liu et al., 2015) and LSUN (Yu et al., 2015) datasets suggest that COCO-GAN is competitive with other state-of-the-art GANs.

Revised: To further demonstrate the effectiveness of COCO-GAN, we perform ablation study in Section 3.3.

Original: Besides the spatial disentanglement property, COCO-GAN is a ﬂexible coordinate-system-aware framework that is suitable for different spatial coordinate systems.
Revised: In Section 3.4, we demonstrate COCO-GAN is a ﬂexible coordinate-system-aware framework that is suitable for different spatial coordinate systems.

Original: Common learning frameworks are conﬁned toCartesian coordinate system.
Revised: Common learning frameworks are conﬁned to

Original: Lacking awareness of coordinate  restricts those frameworks from learning the characteristics of certain image formats.
Revised: [Figure] Cartesian coordinate system, which restricts those frameworks from learning the characteristics of certain image formats.

Original: For instance, panoramas should be[Figure] trained with a cylindrical coordinate system, which has a “cyclic topology” in the horizontal direction.
Revised: For instance, panoramas should be trained with a cylindrical coordinate system, which has a “cyclic topology” in the horizontal direction.

Original: In comparison, COCO-GAN directly learns with a cylindrical coordinate system.
Revised: In comparison, COCO-GAN directly learns with a cylindrical coordinate system.

Original: InFigureand Figure 7, the generated full scenes are naturally cylindrical and continuous while crossing the left and right borders.
Revised: In Figure 6 and Figure 14, the generated full scenes are naturally cylindrical and continuous while crossing the left and right borders.

Original: In addition to COCO-GAN’s capability of learning spatial disentanglement and being ﬂexible to different coordinate systems, we ﬁnd COCO-GAN has multiple interesting applications and characteristics.
Revised: Besides, we ﬁnd COCO-GAN has multiple interesting applications and characteristics.

Original: We select three representative new applications as case studies:“Patch-Inspired Image
Revised: We select three representative new applications as case studies:

Original: Generation”, “Partial-Scene Generation” and “Computation-Friendly Generation”.

Original: “Patch-Inspired Image Generation” takes a real image patch as input.
Revised: Patch-Inspired Image Generation takes a real image patch as input.

Original: COCO-GAN can be inspired by the given patch and further generates a full-image proposal.
Revised: COCO-GAN can be inspired by the given patch and further generates a full-image proposal.

Original: This proposal is partially similar to the given patch, while globally realistic and reasonable.
Revised: This proposal is partially similar to the given patch, while globally realistic and reasonable.

Original: This setting is different from conventional image completion/reconstruction since the information loss during cropping is extreme and the spatial position information is not provided to the model.
Revised: This setting is different from conventional image completion/reconstruction since the information loss during cropping is extreme and the spatial position information is not provided to the model.

Original: Therefore, COCO-GAN has to infer the position of the given patch before generating the whole image.
Revised: Therefore, COCO-GAN has to infer the position of the given patch before generating the whole image.

Original: Further experiments are presented in Section 3.4.
Revised: Further experiments are presented in Section 3.5.

Original: “Partial-Scene Generation” shows that COCO-GAN can generates partial scenes without spending additional computation outside certain designated regions.
Revised: Partial-Scene Generation shows that COCO-GAN can generate partial scenes without spending additional computation outside certain designated regions.

Original: This capability is exclusively beneﬁcial to applications that are only interested in partial information of the full scene.
Revised: This capability is exclusively beneﬁcial to applications that are only interested in partial information of the full scene.

Original: For instance, virtual reality (VR) is only interested in the user’s viewport direction, and COCO-GAN can seamlessly adapt to such viewport-aware settings.
Revised: For instance, virtual reality (VR) is only interested in the user’s viewport direction, and COCO-GAN can seamlessly adapt to such viewport-aware settings.

Original: “Computation-Friendly Generation” demonstrates the computational related merits of COCO-GAN.
Revised: Computation-Friendly Generation demonstrates the computational related merits.

Original: First, since the patches are produced independently, the generator is able to generate patches with high parallelism.
Revised: First, since the patches are produced independently, the generator is able to generate patches with high parallelism.

Original: Second, as the full-image generation is decomposed to patches generation, the minimum memory requirement of generating images can be reduced.
Revised: Second, as the full-image generation is decomposed to patches generation, the minimum memory requirement of generating images can be reduced.

Original: This characteristic enables COCO-GAN to generate images of very high resolution or containing much more complex structures.
Revised: This characteristic enables COCO-GAN to generate images of very high resolution or containing much more complex structures.

Original: In this work, we propose the COCO-GAN , aiming at learning spatial disentanglement with GANs.

Original: Without additional hyper-parameter tuning, both qualitative and quantitative results suggest COCOGAN is competitive with other state-of-the-art GANs in terms of generation quality.

Original: Furthermore,

Original: COCO-GAN is a coordinate-system-aware framework.

Original: We investigate panorama generation with a cylindrical coordinate system, where the generated panoramas have built-in cyclic property in the horizontal direction.

Original: We also investigate three new applications as case studies, including “PatchInspired Image Generation”, “Partial-Scene Generation”, and “Computation-Friendly Generation”.

Original:  COCO-GAN
Revised:  COCO-GAN

Original: Overview.
Revised: Overview.

Original: COCO-GAN consists of three networks: a generator G , a discriminator D , and a auxiliary head Q .
Revised: COCO-GAN consists of three networks: a generator G , a discriminator D , and a auxiliary head Q .

Original: These three networks are trained with four loss terms: patch Wasserstein loss L W , patch gradient penalty loss L GP , spatial consistency loss L S , and content consistency loss L C .
Revised: These three networks are trained with four loss terms: patch Wasserstein loss L W , patch gradient penalty loss L GP , spatial consistency loss L S , and content consistency loss L C .

Original: Compared to conventional GANs that use full images as input for both G and D , COCO-GAN only uses micro patches for G , and macro patches for D .
Revised: Compared to conventional GANs that use full images as input for both G and D , COCO-GAN only uses micro patches for G , and macro patches for D .

Original: The details of micro and macro patches will be described in the paragraph “Spatial coordinate system design”.
Revised: The details of micro and macro patches will be described in the paragraph “Spatial coordinate system design”.

Original: D has two auxiliary prediction heads: the content vector prediction head ( Q ) and the spatial condition prediction head.
Revised: D has two auxiliary prediction heads: the content vector prediction head ( Q ) and the spatial condition prediction head.

Original: Q is trained with an extra optimizer independent to G and D .
Revised: Q is trained with an extra optimizer independent to G and D .

Original: It aims to minimize L C and is used to estimate the original latent vector of the given sample.
Revised: It aims to minimize L C and is used to estimate the original latent vector of the given sample.

Original: The spatial condition prediction head, on the other hand, is jointly trained with the discriminator.
Revised: The spatial condition prediction head, on the other hand, is jointly trained with the discriminator.

Original: It aims at minimizing L S and is used to estimate the macro spatial position of the given sample.
Revised: It aims at minimizing L S and is used to estimate the macro spatial position of the given sample.

Original: Both of the two auxiliary prediction heads are simple feed-forward networks that take inputs from one of the high-level feature maps of the discriminator.
Revised: Both of the two auxiliary prediction heads are simple feed-forward networks that take inputs from one of the high-level feature maps of the discriminator.

Original: COCO-GAN considers two coordinate systems: a micro coordinate system m , which refers to the annotated component that related to the ﬁner coordinate system, on the generator’s side, and a macro coordinate system M , which is related to the coarser macro coordinate system on the discriminator’s side.
Revised: COCO-GAN considers two coordinate systems: a micro coordinate system m , which refers to the annotated component that related to the ﬁner coordinate system, on the generator’s side, and a macro coordinate system M , which is related to the coarser macro coordinate system on the discriminator’s side.

Original: The discriminator learns to distinguish between the generated macro patch ˜ p M and the real macro patch p M .
Revised: The discriminator learns to distinguish between the generated macro patch ˜ p M and the real macro patch p M .

Original: The discriminator also learns to predict two auxiliary outputs: a predicted macro coordinate ˜ c M and a predicted latent vector ˜ z M .
Revised: The discriminator also learns to predict two auxiliary outputs: a predicted macro coordinate ˜ c M and a predicted latent vector ˜ z M .

Original: Outputs from these two networks are then used to compute two auxiliary losses: Spatial Consistency Loss ( L S ) and Content Consistency Loss ( L C ).
Revised: Outputs from these two networks are then used to compute two auxiliary losses: Spatial Consistency Loss ( L S ) and Content Consistency Loss ( L C ).

Original: The objective function of the discriminator D is L W + L GP + L S + L C , the generator G is − L W +
Revised: The objective function of the discriminator D is L W + L GP + L S + L C , the generator G is − L W +

Original: L S + L C , and the content vector prediction head Q is − L W + L C .
Revised: L S + L C , and the content vector prediction head Q is − L W + L C .

Original: Spatial coordinate system.
Revised: Spatial coordinate system.

Original: Before presenting the details of these four loss terms, we introduce our notations ﬁrst.
Revised: Before presenting the details of these four loss terms, we introduce our notations ﬁrst.

Original: We start with designing two spatial coordinate systems, a micro coordinate system C m for the generator G and a macro coordinate system C M for the discriminator D .
Revised: We start with designing two spatial coordinate systems, a micro coordinate system C m for the generator G and a macro coordinate system C M for the discriminator D .

Original: Let
Revised: Let

Original: S N be a space of spatial position sequences, each spatial position sequence s = (cid:104) c mi (cid:105) Ni =1 ∈ S N is an ordered sequence, which c mi ∈ C m .
Revised: S N be a space of spatial position sequences, each spatial position sequence s = (cid:104) c mi (cid:105) Ni =1 ∈ S N is an ordered sequence, which c mi ∈ C m .

Original: During COCO-GAN training, R is some predeﬁned spatial constraints for sampling s from a uniform distribution U ( S N , R ) .
Revised: During COCO-GAN training, R is some predeﬁned spatial constraints for sampling s from a uniform distribution U ( S N , R ) .

Original: The generator G is conditioned by each spatial position c mi , and learns to accordingly produce micro patches ˜ p
Revised: The generator G is conditioned by each spatial position c mi , and learns to accordingly produce micro patches ˜ p

Original: mi = G
Revised: mi = G

Original: ( z | c mi ) .
Revised: ( z | c mi ) .

Original: The (cid:104) ˜
Revised: The (cid:104) ˜

Original: p mi (cid:105) Ni =

Revised: p mi (cid:105) Ni =

Original: = (cid:104) G ( z
Revised: = (cid:104) G ( z

Original: | c mi ) (cid:105)
Revised: | c mi ) (cid:105)

Original: Ni
Revised: Ni

Original: =1 is a sequence of micro patches produced independently while sharing the same latent vector z across the spatial position sequence s .
Revised: =1 is a sequence of micro patches produced independently while sharing the same latent vector z across the spatial position sequence s .

Original: The design of R may need to be slightly changed with respect to the selection of C M and C m .
Revised: The design of R may need to be slightly changed with respect to the selection of C M and C m .

Original: The design principle of R is that the accordingly generated micro patches (cid:104) ˜
Revised: The design principle of R is that the accordingly generated micro patches (cid:104) ˜

Original: p mi (cid:105) Ni
Revised: p mi (cid:105) Ni

Original: =1 should be spatially close to each other.
Revised: =1 should be spatially close to each other.

Original: Then the micro patches are merged by a merging function T ( m → M ) to form a complete macro patch ˜ p M = T ( m → M ) ( (cid:104)
Revised: Then the micro patches are merged by a merging function T ( m → M ) to form a complete macro patch ˜ p M = T ( m → M ) ( (cid:104)

Original: ˜
Revised: ˜

Original: p mi (cid:105) Ni =1 ) as a coarser partial-view of the full-scene ˜ x .
Revised: p mi (cid:105) Ni =1 ) as a coarser partial-view of the full-scene ˜ x .

Original: Meanwhile, we assign ˜ p M with a new spatial position c M under the macro coordinate system for s .
Revised: Meanwhile, we assign ˜ p M with a new spatial position c M under the macro coordinate system for s .

Original: In Figure 1, we illustrate one of the simplest design for the above heuristic functions that we have adopted throughout our experiments.
Revised: In Figure 1, we illustrate one of the simplest design for the above heuristic functions that we have adopted throughout our experiments.

Original: The four micro patches are always a neighbor of each other and can be directly combined into a square macro patch with T ( m → M ) , which is just simple concatenation.
Revised: The four micro patches are always a neighbor of each other and can be directly combined into a square macro patch with T ( m → M ) , which is just simple concatenation.

Original: Figure 3 shows some examples of micro and macro patches generation.
Revised: Figure 3 shows some examples of micro and macro patches generation.

Original: On the real samples side, we also sample s ∼ S , but we directly infer s to macro position c M .
Revised: On the real samples side, we also sample s ∼ S , but we directly infer s to macro position c M .

Original: Then we design a transformation T ( X → M ) with respect to T ( m → M ) to transform a real full-image x to a real macro patch p M by T ( X → M ) ( x | c M ) .
Revised: Then we design a transformation T ( X → M ) with respect to T ( m → M ) to transform a real full-image x to a real macro patch p M by T ( X → M ) ( x | c M ) .

Original: The shape of p M should be the same as ˜ p M .
Revised: The shape of p M should be the same as ˜ p M .

Original: In our simplest experiment setting, T ( X → M ) is a simple cropping function that crops x into p M , which has the same shape as ˜ p M .
Revised: In our simplest experiment setting, T ( X → M ) is a simple cropping function that crops x into p M , which has the same shape as ˜ p M .

Original: During the testing phase, depending on the design of C m , we can directly infer a corresponding spatial position sequence (cid:104)
Revised: During the testing phase, depending on the design of C m , we can directly infer a corresponding spatial position sequence (cid:104)

Original: c m j (cid:105) K j =1 .
Revised: c mj (cid:105) Kj =1 .

Original: It is used to independently produce spatially-disentangled patches that constitute the full-image.
Revised: It is used to independently produce spatially-disentangled patches that constitute the full-image.

Original: Figure 10 demonstrates how the full-image is generated during the testing phase.
Revised: Figure 9 demonstrates how the full-image is generated during the testing phase.

Original: Figure 2a shows some examples of the full-image generation.
Revised: Figure 2a shows some examples of the full-image generation.

Original: Loss functions.
Revised: Loss functions.

Original: The patch Wasserstein loss L W is a patch-level Wasserstein distance loss similar to Wasserstein-GAN (Arjovsky et al., 2017) loss.
Revised: The patch Wasserstein loss L W is a patch-level Wasserstein distance loss similar to Wasserstein-GAN (Arjovsky et al., 2017) loss.

Original: It forces the discriminator to discriminate between the real macro patch p M and the generated macro patch ˜ p M , and on the other hand, encourages the generator to confuse the discriminator with seemingly realistic ˜ p M .
Revised: It forces the discriminator to discriminate between the real macro patch p M and the generated macro patch ˜ p M , and on the other hand, encourages the generator to confuse the discriminator with seemingly realistic ˜ p M .

Original: Its complete form is
Revised: Its complete form is

Original: [Equation]
Revised: [Equation]

Original: where x ∼ P r and (cid:104) c mi (cid:105) Ni =1 ∼ U ( S N , R ) .
Revised: where x ∼ P r and (cid:104) c mi (cid:105) Ni =1 ∼ U ( S N , R ) .

Original: Note that P r is the real data distribution.
Revised: Note that P r is the real data distribution.

Original: We also apply
Revised: We also apply

Original: Gradient Penalty (Gulrajani et al., 2017) to the patches generation:
Revised: Gradient Penalty (Gulrajani et al., 2017) to the patches generation:

Original: [Equation]
Revised: [Equation]

Original: where ˜ p M ∼ P g .
Revised: where ˜ p M ∼ P g .

Original: Note that P g is the generator distribution.
Revised: Note that P g is the generator distribution.

Original: The spatial consistency loss L S is similar to ACGAN loss (Odena et al., 2017).
Revised: The spatial consistency loss L S is similar to ACGAN loss (Odena et al., 2017).

Original: A slight difference is that c mi has relatively more continuous values than the discrete setting of ACGAN.
Revised: A slight difference is that c mi has relatively more continuous values than the discrete setting of ACGAN.

Original: As a result, we apply a distance measurement loss for L S , which is an L 2 -loss between ˜ c M and c M .
Revised: As a result, we apply a distance measurement loss for L S , which is an L 2 -loss between ˜ c M and c M .

Original: It aims to train the generator of COCO-GAN to generate the corresponding micro patch by G ( z | c mi ) with respect to the given spatial condition c mi .
Revised: It aims to train the generator of COCO-GAN to generate the corresponding micro patch by G ( z | c mi ) with respect to the given spatial condition c mi .

Original: The spatial consistency loss is
Revised: The spatial consistency loss is

Original: [Equation]
Revised: [Equation]

Original: On the other hand, the content consistency loss L C is similar to a hybrid of infoGAN loss (Chen et al., 2016) and the latent space constraint loss (Chang et al., 2018).
Revised: On the other hand, the content consistency loss L C is similar to a hybrid of infoGAN loss (Chen et al., 2016) and the latent space constraint loss (Chang et al., 2018).

Original: The former one uses a separate optimizer to optimize an auxiliary network Q , which aims to reconstruct the original latent vector.
Revised: The former one uses a separate optimizer to optimize an auxiliary network Q , which aims to reconstruct the original latent vector.

Original: The latter one suggests that the latent space consistency loss can be a distance measurement instead of minimizing the KL-divergence as the original infoGAN does.
Revised: The latter one suggests that the latent space consistency loss can be a distance measurement instead of minimizing the KL-divergence as the original infoGAN does.

Original: In our experiments, we train the extra Q network with a separate optimizer and directly minimize L 1 -loss between ˜ z and z .
Revised: In our experiments, we train the extra Q network with a separate optimizer and directly minimize L 1 -loss between ˜ z and z .

Original: L C aims to force the generator to produce shared context between patches that share the same latent vector but locate at different micro coordinate positions.
Revised: L C aims to force the generator to produce shared context between patches that share the same latent vector but locate at different micro coordinate positions.

Original: The content consistency loss is deﬁned by
Revised: The content consistency loss is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: To ensure that Z , C m , and C M are all having a similar scale, we directly concatenate and feedthem to the generator.
Revised: To ensure that Z , C m , and C M share the similar scale, which are directly concatenated and feed to G .

Original: We evaluate the maximum possible pixel position of C m and C M , then normalize the range into [ − 1 , 1] .
Revised: We evaluate the maximum possible pixel position of C m and C M , then normalize the range into [ − 1 , 1] .

Original: For the latent space Z , althoughusing uniform sampling between [ − 1 , 1] should be numerically more compatible with the normalized spatial condition space, we empirically do not observe signiﬁcant differences even if we switch to random sampling with a zero-mean and unit-variance normal distribution.
Revised: For the latent space Z , although uniform sampling between [ − 1 , 1] should be numerically more compatible with the normalized spatial condition space, we empirically do not observe significant differences even if we switch to random sampling with a zero-mean and unit-variance normal distribution.

Original: For simplicity, we adopt uniform sampling strategy throughout our experiments.
Revised: For simplicity, we adopt uniform sampling strategy throughout our experiments.

Original: Training details.
Revised: Training details.

Original: Our generator and discriminator architectures follow the idea of projection discriminator (Miyato & Koyama, 2018), both with ResNet (He et al., 2016) based architecture and adding class-projection to the discriminator.
Revised: Our generator and discriminator architectures follow the idea of projection discriminator (Miyato & Koyama, 2018), both with ResNet (He et al., 2016) based architecture and adding class-projection to the discriminator.

Original: All convolutional and feed-forward layers of generator and discriminator are added with the spectral-normalization scheme (Miyato et al., 2018) as suggested in (Zhang et al., 2018).
Revised: All convolutional and feed-forward layers of generator and discriminator are added with the spectral-normalization scheme (Miyato et al., 2018) as suggested in (Zhang et al., 2018).

Original: A more detailed architecture diagram is illustrated in Appendix B.
Revised: A more detailed architecture diagram is illustrated in Appendix B.

Original: We also add conditional-batch-normalization (CBN) (Dumoulin et al., 2016) to the generator.
Revised: We also add conditional-batch-normalization (CBN) (Dumoulin et al., 2016) to the generator.

Original: In our design, CBN is conditioned on the given spatial positions and the input latent vector.
Revised: In our design, CBN is conditioned on the given spatial positions and the input latent vector.

Original: It learns to normalize the feature maps with respect to the given conditions.
Revised: It learns to normalize the feature maps with respect to the given conditions.

Original: However, our implementation has a crucial difference from the one described in (Miyato & Koyama, 2018): our spatial positions are real values rather than discrete classes.
Revised: However, our implementation has a crucial difference from the one described in (Miyato & Koyama, 2018): our spatial positions are real values rather than discrete classes.

Original: We alternatively adopt similar strategy to (de Vries et al., 2017) with a slight modiﬁcation.
Revised: We alternatively adopt similar strategy to (de Vries et al., 2017) with a slight modiﬁcation.

Original: Instead of using MLPs to produce ∆ γ and ∆ β , we make MLPs directly output γ and β .
Revised: Instead of using MLPs to produce ∆ γ and ∆ β , we make MLPs directly output γ and β .

Original: For a K -channel input feature map i K with mean recorder µ K and variance recorder σ K , we creates two learnable MLP layers, MLP γ and MLP β .
Revised: For a K -channel input feature map i K with mean recorder µ K and variance recorder σ K , we creates two learnable MLP layers, MLP γ and MLP β .

Original: The output feature map o M is calculated as 
Revised: The output feature map o M is calculated as o M =

Original: [Equation]
Revised: (( i K − µ K ) /σ K ) ∗ MLP γ ( i K ) + MLP β ( i K ) .

Original: We use Adam (Kingma & Ba, 2014) optimizer with β 1 = 0 and β 2 = 0 .
Revised: We use Adam (Kingma & Ba, 2014) optimizer with β 1 = 0 and β 2 = 0 .

Original: 999 for both the generator and the discriminator.
Revised: 999 for both the generator and the discriminator.

Original: The learning rates are based on the Two Time-scale Update Rule (TTUR) (Heusel et al., 2017), setting 0 .
Revised: The learning rates are based on the Two Time-scale Update Rule (TTUR) (Heusel et al., 2017), setting 0 .

Original: for the generator and 0 . 

Revised: for the generator and 0 .

Original: for the discriminator.
Revised: for the discriminator.

Original: We do not speciﬁcally balance the generator and the discriminator by manually setting how many iterations to update the generator once as described in the WGAN paper (Arjovsky et al., 2017).[Figure] [Figure]
Revised: We do not speciﬁcally balance the generator and the discriminator by manually setting how many iterations to update the generator once as described in the WGAN paper (Arjovsky et al., 2017).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Although COCO-GAN framework supports the spatial positions to be uniformly and continuously sampled within normalized range [ − 1 , 1] , we empirically ﬁnd that only sampling the discrete spatial positions that is used to inference will result in better generation quality.
Revised: Although COCO-GAN framework supports the spatial positions to be uniformly and continuously sampled within normalized range [ − 1 , 1] , we empirically ﬁnd that only sampling the discrete spatial positions that is used to inference will result in better generation quality.

Original: For instance, if the full-image is formed by concatenating four micro patches on each axis, the model only uniformly samples spatial positions from the set of four discrete points, {− 1 , − 1 / 3 , 1 / 3 , 1 } on each axis of the coordinate system.
Revised: For instance, if the full-image is formed by concatenating four micro patches on each axis, the model only uniformly samples spatial positions from the set of four discrete points, {− 1 , − 1 / 3 , 1 / 3 , 1 } on each axis of the coordinate system.

Original: We adopt this uniform and discrete sampling strategy throughout the experiments.
Revised: We adopt this uniform and discrete sampling strategy throughout the experiments.

Original: The root cause of degradation in generation quality is still unclear.
Revised: The root cause of degradation in generation quality is still unclear.

Original: One possible hypothesis is that the task difﬁculty dramatically increases while sampling withuniform and continuous spatial positions.
Revised: One possible hypothesis is that the task difﬁculty dramatically increases while sampling with continuous spatial positions.

Original: We ﬂag further analysis and solution toward this phenomenon as an important future research direction.
Revised: We ﬂag further analysis and solution toward this phenomenon as an important future research direction.

Revised: Some comparisons between continuous sampling and discrete sampling are shown in Appendix F.

Revised: [Figure]

Revised: Figure 2:

Revised: Without any post-processing, the generated full-images of COCO-GAN are visually smooth and globally coherent.

Revised: More full-image generation results are shown in Figure

Original:  3. Q UALITY OF G ENERATED I MAGES
Revised: [Figure] [Table] 3. Q UALITY OF G ENERATED I MAGES

Original: We start with validating COCO-GAN on CelebA (Liu et al., 2015) and the bedroom category of
Revised: We start with validating COCO-GAN on CelebA (Liu et al., 2015) and the bedroom category of

Original: LSUN (Yu et al., 2015).
Revised: LSUN (Yu et al., 2015).

Original: For CelebA dataset, the resolutions of full-image, micro patch, and macro patch are 128 × 128 , 32 × 32 , and 64 × 64 , respectively.
Revised: For CelebA dataset, the resolutions of full-image, micro patch [Table] and macro patch are 128 × 128 , 32 × [Table]

Original:  We choose 32 × 32 for micro patches in this experiment since smaller patch size would be too small for the model (neither for human) to observe useful information.
Revised: and 64 × 64 , respectively. We choose 32 × 32 for micro patches in this experiment since smaller patch size would be too small for the model (neither for human) to observe useful information.

Original: On the other hand, larger patch size makes macro patch size too similar to the full-image size, which is hard to demonstrate the idea of the discriminator can learn without access to the full-image.
Revised: On the other hand, larger patch size makes macro patch size too similar to the full-image size, which is hard to demonstrate the idea of COCO-GAN can learn without access to the full-image.

Original: For LSUN dataset, the full-image is with 256 × 256 resolution, micro patches with 64 × 64 resolution and macro patches with 128 × 128 resolution.
Revised: For LSUN dataset, the full-image is with 256 × 256 resolution, micro patches with 64 × 64 resolution and macro patches with 128 × 128 resolution.

Original: We choose 64 × 64 for micro patch sizein this experiment since the micro patch to full-image ratio is the same with the CelebA experiment.
Revised: We choose 64 × 64 for micro patch size since the micro patch to full-image ratio is the same with the CelebA experiment.

Original: We report Frchet Inception Distance (FID) (Heusel et al., 2017) in Table 1 as quantitative comparisons with state-of-the-art GANs.
Revised: We report Frchet Inception Distance (FID) (Heusel et al., 2017) in Table 1 as quantitative comparisons with state-of-the-art GANs.

Original: Since many other state-of-the-art models do not use full-resolution of the datasets, we accordingly run COCO-GAN in different resolutions without changing hyperparameters other than input size and micro/macro patch size.
Revised: Since many other state-of-the-art models do not use full-resolution of the datasets, we accordingly run COCO-GAN in different resolutions without changing hyperparameters other than input size and micro/macro patch size.

Original: Throughout these experiments, we choose to always retain the micro patch size to be 1/16 (1/4 for height and 1/4 for width) of the fullimage size and macro patch size to be 1/4 (1/2 for height and 1/2 for width) of the full-image size.
Revised: Throughout these experiments, we choose to always retain the micro patch size to be 1/16 (1/4 for height and 1/4 for width) of the fullimage size and macro patch size to be 1/4 (1/2 for height and 1/2 for width) of the full-image size.

Original: Without additional hyper-parameter tuning, the results suggest COCO-GAN is both qualitatively and quantitatively competitive with other state-of-the-art GANs.
Revised: Without additional hyper-parameter tuning, the results suggest COCO-GAN is both qualitatively and quantitatively competitive with other state-of-the-art GANs.

Original: In Appendix F, we also provide Wasserstein distance and FID score through time as training indicators.
Revised: In Appendix H, we also provide Wasserstein distance and FID score through time as training indicators.

Original: The curves suggest that COCO-GAN is stable during training.
Revised: The curves suggest that COCO-GAN is stable during training.

Original: [Table]

Original: 3. L ATENT S PACE C ONTINUITY
Revised: 3. L ATENT S PACE C ONTINUITY

Original: To demonstrate more precisely the space continuity of COCO-GAN , we perform the interpolation experiment in three directions: micro patches interpolation, spatial positions interpolation, and fullimages interpolation. 
Revised: To demonstrate more precisely the space continuity, we perform the interpolation experiment in three directions: micro patches interpolation, spatial positions interpolation, and full-images interpolation. [Figure]

Original: • Micro Patches Interpolation
Revised: Micro Patches Interpolation.

Original: The simplest interpolation experiment is the in-class ( e g . ﬁxed spatial condition) interpolation between latent vectors.
Revised: The simplest interpolation experiment is the in-class ( e.g . ﬁxed spatial condition) interpolation between latent vectors.

Original: With a ﬁxed spatial position c microi , we randomly sample two latent vectors z 1 and z 2 .
Revised: With a ﬁxed spatial position c microi , we randomly sample two latent vectors z 1 and z 2 .

Original: Then perform interpolation between z 1 and z 2 through a slerppath (White, 2016).
Revised: Then perform interpolation between z 1 and z 2 through a slerp-path (White, 2016).

Original: The results in Figure 4a suggest that for each of the spatial position, the latent space Z has continuity.
Revised: The results in Figure 5a suggest that for each of the spatial position, the latent space Z has continuity.

Original: • Spatial Positions Interpolation Another simple interpolation experiment is inter-class ( e g . between classes) interpolation with a ﬁxed latent vector.
Revised: Another simple interpolation experiment is inter-class ( e.g . between classes) interpolation with a ﬁxed latent vector.

Original: We directly linearly-interpolate spatial position between [ − 1 , 1] when the latent vector z is ﬁxed.
Revised: We directly linearly-interpolate spatial position between [ − 1 , 1] when the latent vector z is ﬁxed.

Original: The results in Figure 5 show that, although we only uniformly sample spatial positions within a discrete spatial position set, the spatial position interpolation is still continuous.
Revised: The results in Figure 4 show that, although we only uniformly sample spatial positions within a discrete spatial position set, the spatial position interpolation is still continuous.

Original: An interesting observation is about the interpolation at the position between the eyebrows.
Revised: An interesting observation is about the interpolation at the position between the eyebrows.

Original: In this example, COCO-GAN

Original:  does not know the existence of the smooth area glabella) between two eyes due to the discrete and sparse spatial positions sampling strategy.
Revised: In this example, COCO-GAN does not know the existence of the smooth area (glabella) between two eyes due to the discrete and sparse spatial positions sampling strategy.

Original: Instead, it learns to directly deform the shape of eye to switch from one eye to another.
Revised: Instead, it learns to directly deform the shape of eye to switch from one eye to another.

Original: This phenomenon raises an interesting discussion, even the model learns to produce high-quality face images, it still may learn wrong relationship of objects behind the scene.
Revised: This phenomenon raises an interesting discussion, even the model learns to produce high-quality face images, it still may learn wrong relationship of objects behind the scene.

Original: • Full-Images Interpolation
Revised: Full-Images Interpolation.

Original: The hardest interpolation is to directly interpolate full-images between two latent vectors.
Revised: The hardest interpolation is to directly interpolate full-images between two latent vectors.

Original: All micro patches generated with different spatial positions must all change synchronously to make the full-image interpolation smooth.
Revised: All micro patches generated with different spatial positions must all change synchronously to make the full-image interpolation smooth.

Original: We randomly sample two latent vectors zand z 2 .
Revised: We randomly sample two latent vectors z 1 and z 2 .

Original: With any given interpolation point z (cid:48) in the slerp path between z 1 and z 2 , the generator uses the full spatial position sequence (cid:104)
Revised: With any given interpolation point z (cid:48) in the slerp path between z 1 and z 2 , the generator uses the full spatial position sequence (cid:104)

Original: c mj (cid:105)
Revised: c mj (cid:105)

Original: Kj
Revised: Kj

Original: =1 to generate all corresponding patches.
Revised: = to generate all corresponding patches.

Original: Then we merge all generated micro patches with T ( m → M ) and forms a full-image x (cid:48) .
Revised: Then we merge all generated micro patches with T ( m → M ) and forms a full-image x (cid:48) .

Original: The interpolation results in Figure 4a and Figure 4b show that all micro patches can interpolate smoothly and synchronously.
Revised: The interpolation results in Figure 5a and Figure 5b show that all micro patches can interpolate smoothly and synchronously.

Original: This result suggests that COCO-GAN learns the main latent space Z as well as the correlation between micro patches, and the spatial conditions C m are disentangled. 
Revised: This result suggests that COCO-GAN learns the main latent space Z as well as the correlation between micro patches, and the spatial conditions C m are disentangled. 3. A BLATION S TUDY

Revised: The ablation study is conducted in two folds: we ﬁrst show that a straightforward approach fails in

Revised: COCO-GAN setting, then we study the trade-offs of each component of COCO-GAN.

Original: 3. P ANORAMA G ENERATION AND P ARTIAL S CENE G ENERATION
Revised: A straightforward approach. One straightforward approach to the learning and inference with partial views is using a full-sized generator and a patch discriminator, and then pre-calculating the locations of feature maps that are associated with a speciﬁc partial view. We refer to this method as M afterward.

Revised: Despite M still implicitly uses conditional coordinating in feature map selection, we observe that it fails to generate high-quality samples in comparison with COCO-GAN.

Revised: We show the

Revised: FID score results in Table 2.

Revised: More experimental details are shown in Figure 15 and some generated samples in Figure 16.

Revised: [Table]

Revised: The trade-offs of each component.

Revised: We perform ablation study in CelebA 64 × 64 setting with ﬁve conﬁgurations: “continuous sampling” demonstrates that using continuous uniform sampling strategy for spatial positions during training will result in moderate generation quality drop; “optimal D ” lets the discriminator directly discriminate the full image while the generator still generates micro patches; “optimal G ” lets the generator directly generate the full image while the discriminator still discriminates macro patches; “without

Revised: Q ” removes the Q network from COCO-GAN; “multiple G ” trains an individual generator for each spatial position.

Revised: The results in Table 2 suggest Q network is not a necessary component if not considering the “Patch-Inspired

Revised: Image Generation” application.

Revised: Surprisingly, despite the convergence speed is different, “optimal discriminator”, COCO-GAN, and “optimal generator” (ordered by convergence speed from fast to slow) can all achieve similar FID scores if with sufﬁcient training time.

Revised: The difference in convergence speed is expected, since “optimal discriminator” provides the generator with more accurate and global adversarial loss.

Revised: In contrast, the “optimal generator“ has relatively more parameters and layers to optimize, which causes the convergence speed slower than COCO-GAN.

Revised: Lastly, the “multiple generators” setting cannot converge well.

Revised: Although it can also concatenate micro patches without obvious seams as COCO-GAN does, the full-image results often cannot agree and are not globally coherent.

Revised: More experimental details and generated samples are shown in Figure 17 and Figure 18.

Revised: 3. P ANORAMA G ENERATION AND P ARTIAL S CENE G ENERATION

Original: Generating panoramas using GANs is an interesting problem but has never been carefully investigated.
Revised: Generating panoramas using GANs is an interesting problem but has never been carefully investigated.

Original: Different from simple image generation, panoramas are expected to be cylindrical and cyclic in the horizontal direction.
Revised: Different from simple image generation, panoramas are expected to be cylindrical and cyclic in the horizontal direction.

Original: However, normal GANs do not have built-in ability to handle such cyclic characteristic if without special types of padding mechanism support (Cheng et al., 2018).
Revised: However, normal GANs do not have built-in ability to handle such cyclic characteristic if without special types of padding mechanism support (Cheng et al., 2018).

Original: In contrast, COCO-GAN is a coordinate-system-aware learning framework.
Revised: In contrast, COCO-GAN is a coordinate-system-aware learning framework.

Original: We can easily adapt a[Figure] [Figure] cylindrical coordinate system, and generate panoramas that are cyclic in the horizontal direction as shown in Figure 6 and Figure 7.
Revised: We can easily adapt a cylindrical coordinate system, and generate panoramas that are cyclic in the horizontal direction as shown in Figure 6 and Figure 14.

Original: To train COCO-GAN with a panorama dataset under a cylindrical coordinate system, the spatial position sampling strategy needs to be slightly modiﬁed.
Revised: To train COCO-GAN with a panorama dataset under a cylindrical coordinate system, the spatial position sampling strategy needs to be slightly modiﬁed.

Original: In the horizontal direction, the sampled value within the normalized range [ − 1 , 1] is treated as an angular value θ , and then is projected with cos( θ ) and sin( θ ) individually to form a unit-circle on a 2D surface.
Revised: In the horizontal direction, the sampled value within the normalized range [ − 1 , 1] is treated as an angular value θ , and then is projected with cos( θ ) and sin( θ ) individually to form a unit-circle on a 2D surface.

Original: Along with the normal sampling on the vertical axis, a cylindrical coordinate system is formed.
Revised: Along with the normal sampling on the vertical axis, a cylindrical coordinate system is formed.

Original: We ﬁrst take the sky-box format of Matterport3D (Chang et al., 2017) dataset to obtain panoramas for training and testing.
Revised: We ﬁrst take the sky-box format of Matterport3D (Chang et al., 2017) dataset to obtain panoramas for training and testing.

Original: The sky-boxes consist of six faces of a 3D cube.
Revised: The sky-boxes consist of six faces of a 3D cube.

Original: We preprocess and project the sky-box to a cylinder using Mercator projection, the resulting cylindrical image size is 768 × 512 .
Revised: We preprocess and project the sky-box to a cylinder using Mercator projection, the resulting cylindrical image size is 768 × 512 .

Original: Since the Mercator projection creates extreme sparsity near the northern and southern poles, which lacks information, we directly remove the upper and lower 1 / 4 areas.
Revised: Since the Mercator projection creates extreme sparsity near the northern and southern poles, which lacks information, we directly remove the upper and lower 1 / 4 areas.

Original: Eventually, the size of panorama we used for training is 768 × 256 pixels.
Revised: Eventually, the size of panorama we used for training is 768 × 256 pixels.

Original: We also ﬁnd COCO-GAN has interesting connection with virtual reality (VR).
Revised: We also ﬁnd COCO-GAN has interesting connection with virtual reality (VR).

Original: VR is known to have tight computational budget due to high frame-rate requirement and high resolution demand.
Revised: VR is known to have tight computational budget due to high frame-rate requirement and high resolution demand.

Original: It is hard to generate full-scene for VR in real time using standard generative models.
Revised: It is hard to generate full-scene for VR in real time using standard generative models.

Original: Some recent VR studies on omnidirectional view rendering and streaming (Corbillon et al., 2017b; Ozcinar et al.,  2017; Corbillon et al., 2017a) are focusing on reducing computational cost or network bandwidth by adapting the user’s viewport.
Revised: Some recent VR studies on omnidirectional view rendering and streaming (Corbillon et al., 2017b; Ozcinar et al., [Figure] 2017; Corbillon et al., 2017a) are focusing on reducing computational cost or network bandwidth by adapting the user’s viewport.

Original: COCO-GAN can easily inherit the same strategy and achieve userviewport-aware partial-scene generation based on its effectiveness in spatial disentanglement and panorama generation.
Revised: COCO-GAN can easily inherit the same strategy and achieve userviewport-aware partial-scene generation based on its effectiveness in spatial disentanglement and panorama generation.

Original: This can largely reduce unnecessary computational cost outside the region of interest, thus making image generation in VR more applicable.
Revised: This can largely reduce unnecessary computational cost outside the region of interest, thus making image generation in VR more applicable.

Original: 3. P ATCH -I NSPIRED I MAGE G ENERATION
Revised: 3. P ATCH -I NSPIRED I MAGE G ENERATION

Original: The content consistency loss equips the discriminator with the ability to approximate the original latent vector of generated macro patch ˜ p macro using the discriminator’s auxiliary content vector prediction ˜ z .
Revised: The content consistency loss equips the discriminator with the ability to approximate the original latent vector of generated macro patch ˜ p macro using the discriminator’s auxiliary content vector prediction ˜ z .

Original: This property can also generalize to any real macro patch p macro .
Revised: This property can also generalize to any real macro patch p macro .

Original: The approximation[Figure] process does not require a spatial position; the spatial position is implicitly inferred by the spatial position prediction ˜ c M .
Revised: The approximation process does not require a spatial position; the spatial position is implicitly inferred by the spatial position prediction ˜ c M .

Original: The generator can accordingly generate a full-image x (cid:48) with respect to ˜ z .
Revised: The generator can accordingly generate a full-image x (cid:48) with respect to ˜ z .

Original: The generated x (cid:48) should be partially similar to the given macro patch and also be globally coherent.
Revised: The generated x (cid:48) should be partially similar to the given macro patch and also be globally coherent.

Original: Such x (cid:48) can be seen as a generated sample inspired by the given macro patch.
Revised: Such x (cid:48) can be seen as a generated sample inspired by the given macro patch.

Original: One important footnote is that most of the information of a real full-image is lost while retrieving p M .
Revised: One important footnote is that most of the information of a real full-image is lost while retrieving p M .

Original: As a result, the produced guess of full-image is not guaranteed to be identical to the original real image.
Revised: As a result, the produced guess of full-image is not guaranteed to be identical to the original real image.

Original: We provides some examples of patch-inspired image generation in Figure 8.
Revised: We provides some examples of patch-inspired image generation in Figure 7.

Original: The results show that x (cid:48) can loosely retain some local structure or global characteristic of the original image, such as gender, face direction, and facial expression.
Revised: The results show that x (cid:48) can loosely retain some local structure or global characteristic of the original image, such as gender, face direction, and facial expression.

Original: This process is also similar to image inpainting  except two key differences.
Revised: This process is also similar to image inpainting (Liu et al., 2018a; Yeh et al., 2017; Yang et al., 2017) except two key differences.

Original: First, the spatial position of the macro patch is not explicitly given.
Revised: First, the spatial position of the macro patch is not explicitly given.

Original: Existing image inpainting frameworks assume  thespatial position of remaining parts of the image is known, while COCO-GAN can infer by itself.
Revised: Existing image inpainting frameworks assume that the remaining parts of the image are already at their optimal positions, whereas COCO-GAN can infer by itself.

Revised: Take human face for instance, if only given a cropped patch of a face image without further providing the position of the patch in the original image, COCO-GAN can still infer the position of the patch and reconstruct a full face image, while common inpainting frameworks may not reconstruct a correctly structured and well centered human face.

Original: Second, most inpainting frameworks do not assume the image is extremely damaged, like loosing 75% of information in our examples.
Revised: Second, most inpainting frameworks do not assume the image is extremely damaged, like loosing 75% of information in our examples.

Original: In Figure 9, we accordingly compare COCO-GAN with partial convolution (Liu et al., 2018a), which is one of the state-of-the-art image inpainting methods.
Revised: In Figure 8, we accordingly compare COCO-GAN with partial convolution (Liu et al., 2018a), which is one of the state-of-theart image inpainting methods.

Original: For the partial convolution method, we simply place the macro patch at the center since the spatial position of the macro patch is unknown.
Revised: For the partial convolution method, we simply place the macro patch at the center since the spatial position of the macro patch is unknown.

Original: The results show that, unlike COCO-GAN , the partial convolution method (Liu et al., 2018a) cannot handle this situation well.
Revised: The results show that, unlike COCO-GAN , the partial convolution method (Liu et al., 2018a) cannot handle this situation well.

Original: 3. C OMPUTATION -F RIENDLY G ENERATION
Revised: 3. C OMPUTATION -F RIENDLY G ENERATION

Original: Recent studies in high-resolution image generation (Karras et al., 2017; Mescheder et al., 2018) have gained lots of success.
Revised: Recent studies in high-resolution image generation (Karras et al., 2017; Mescheder et al., 2018) have gained lots of success.

Original: We observe a shared conundrum of these existing works is the memory requirement.
Revised: We observe a shared conundrum of these existing works is the memory requirement.

Original: They usually require some workarounds to improve memory consumption during training, such as decreasing the batch size (Karras et al., 2017) or cutting down the number of feature maps (Mescheder et al., 2018).
Revised: They usually require some workarounds to improve memory consumption during training, such as decreasing the batch size (Karras et al., 2017) or cutting down the number of feature maps (Mescheder et al., 2018).

Original: The memory requirement problem cannot be easily resolved without speciﬁc hardware support, which makes the generation of over 1024 × 1024 resolution images hard to achieve.
Revised: The memory requirement problem cannot be easily resolved without speciﬁc hardware support, which makes the generation of over 1024 × 1024 resolution images hard to achieve.

Original: These types of high-resolution images are commonly seen in panoramas, street views, and medical images.
Revised: These types of high-resolution images are commonly seen in panoramas, street views, and medical images.

Original: [Figure]
Revised: [Figure]

Original: Since the information loss of the cropping process (
Revised: Since the information loss of the cropping process (

Original: (a) to (b)) is critical, we do not expect (a)and (d) to be identical.
Revised: (a) to b)) is critical, we do not expect (a) to be identical.

Original: Instead, (b) and (c) should be visually similar and (d) should be globally coherent.
Revised: Instead, (b) and (c) should be visually similar and (d) should be globally coherent.

Original: More examples are shown in Appendix E.
Revised: More examples are shown in Appendix G.

Original: [Figure] In contrast, COCO-GANhas spatial disentanglement, which only requires partial views of the full-image for both training and inference.
Revised: In contrast, COCO-GAN only requires partial views of the full-image for both training and inference.

Original: This characteristic largely reduces the minimum memory requirement while dealing with high-resolution images.
Revised: This characteristic largely reduces the minimum memory requirement while dealing with high-resolution images.

Original: In Section 3.1, we show that COCO-GAN can generate and compose a 128 × 128 image with patches of 32 × 32 resolution.
Revised: In Section 3.1, we show that COCO-GAN can generate and compose a 128 × 128 image with patches of 32 × 32 resolution.

Original: In other words, we can train a CIFAR-10-sized model to generate 128 × 128 resolution images while the results are still high quality.
Revised: In other words, we can train a CIFAR-10-sized model to generate 128 × 128 resolution images while the results are still high quality.

Original: The characteristic of spatial disentanglement is the ﬁrst step toward super-highresolution image generation with limited memory resource.
Revised: The characteristic of spatial disentanglement is the ﬁrst step toward super-high-resolution image generation with limited memory resource.

Original: Moreover, some state-of-the-art structures of deep models tend to require more parameters (if without reducing number of channels), such as the skip-connection structure (He et al., 2016) used in projection discriminator (Miyato & Koyama, 2018), inception (Szegedy et al., 2015), and self-attention (Zhang et al., 2018).
Revised: Moreover, some state-of-the-art structures of deep models tend to require more parameters (if without reducing number of channels), such as the skip-connection structure (He et al., 2016) used in projection discriminator (Miyato & Koyama, 2018), inception (Szegedy et al., 2015), and self-attention (Zhang et al., 2018).

Original: COCO-GAN is able to equip many of these complex structures since our COCO-GAN model is relatively light-weight with a smaller receptive ﬁeld and a shallower structure. 
Revised: COCO-GAN is able to equip many of these complex structures since our COCO-GAN model is relatively light-weight with a smaller receptive ﬁeld and a shallower structure. [Figure]

Revised: Patch-inspired image generation is globally more coherent than the partial convolution method.

Original: Furthermore, the spatial disentanglement makes the generation of micro patches independent after the latent vector and the spatial positions are decided.
Revised: Furthermore, the spatial disentanglement makes the generation of micro patches independent after the latent vector and the spatial positions are decided.

Original: This characteristic enables the generation of micro patches to have high parallelism and take advantage of modern computation architectures.
Revised: This characteristic enables the generation of micro patches to have high parallelism and take advantage of modern computation architectures.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generative Adversarial Network (GAN) (Goodfellow et al., 2014) has shown its potential and ﬂexibility to many different tasks.
Revised: Generative Adversarial Network (GAN) (Goodfellow et al., 2014) has shown its potential and ﬂexibility to many different tasks.

Original: Recent studies on GANs are focusing on generating highresolution and high-quality synthetic images in different settings.
Revised: Recent studies on GANs are focusing on generating high-resolution and high-quality synthetic images in different settings.

Original: For instance, generating images with 1024 × 1024 resolution (Karras et al., 2017; Mescheder et al., 2018), generating images with low-quality synthetic images as condition (Shrivastava et al., 2017), and by applying segmentation map as conditions (Wang et al., 2017).
Revised: For instance, generating images with 1024 × 1024 resolution (Karras et al., 2017; Mescheder et al., 2018), generating images with low-quality synthetic images as condition (Shrivastava et al., 2017), and by applying segmentation map as conditions (Wang et al., 2017).

Original: However, these prior work share the similar assumptions: the model must access and generate the full-image in a single shot.
Revised: However, these prior work share the similar assumptions: the model must access and generate the full-image in a single shot.

Original: This assumption consumes an unavoidable and signiﬁcant amount of memory when the size of the targeting image is relatively large, and therefore making it difﬁcult to satisfy memory requirements for both training and inference.
Revised: This assumption consumes an unavoidable and signiﬁcant amount of memory when the size of the targeting image is relatively large, and therefore making it difﬁcult to satisfy memory requirements for both training and inference.

Original: Searching for a solution towards this problem is one of the initial motivations of this work.
Revised: Searching for a solution towards this problem is one of the initial motivations of this work.

Original: COCO-GAN shares some similarities to Pixel-RNN (van den Oord et al., 2016), which is a pixellevel generation framework while COCO-GAN is a patch-level generation framework.
Revised: COCO-GAN shares some similarities to Pixel-RNN (van den Oord et al., 2016), which is a pixellevel generation framework while COCO-GAN is a patch-level generation framework.

Original: Pixel-RNN transforms the image generation task into a sequence generation task, maximizes the log-likelihood directly.
Revised: Pixel-RNN transforms the image generation task into a sequence generation task, maximizes the log-likelihood directly.

Original: In contrast, COCO-GAN aims at disentangling the spatial dependencies between micro patches.
Revised: In contrast, COCO-GAN aims at disentangling the spatial dependencies between micro patches.

Original: It also utilizes the adversarial loss to ensure smoothness between adjacent micro patches.
Revised: It also utilizes the adversarial loss to ensure smoothness between adjacent micro patches.

Original: CoordConv (Liu et al., 2018b) is another similar work but with fundamental differences.
Revised: CoordConv (Liu et al., 2018b) is another similar work but with fundamental differences.

Original: CoordConv provides spatial positioning information directly to the convolutional kernels in order to the coordinate transform problem and shows multiple improvements in different tasks.
Revised: CoordConv provides spatial positioning information directly to the convolutional kernels in order to the coordinate transform problem and shows multiple improvements in different tasks.

Original: In contrast,
Revised: In contrast,

Original: COCO-GAN uses spatial conditions as an input condition of the generator and an auxiliary output of the discriminator.
Revised: COCO-GAN uses spatial conditions as an input condition of the generator and an auxiliary output of the discriminator.

Original: This setting enforces both the generator and the discriminator to learn spatial disentanglement and correlations between the generated micro patches.
Revised: This setting enforces both the generator and the discriminator to learn coordinating and correlations between the generated micro patches.

Original: We have also considered incorporating CoordConv into COCO-GAN .
Revised: We have also considered incorporating CoordConv into COCO-GAN .

Original: However, empirical results show little visual improvement.
Revised: However, empirical results show little visual improvement.

Original: Group convolution (Krizhevsky et al., 2012) is another work that is highly related to COCO-GAN .
Revised: Group convolution (Krizhevsky et al., 2012) is another work that is highly related to COCO-GAN .

Original: While group convolution aims at reducing computational costs by disentangling channels inside a convolution layer, our model learns to disentangle on the spatial level and is highly parallelizable.
Revised: While group convolution aims at reducing computational costs by disentangling channels inside a convolution layer, our model learns to disentangle on the spatial level and is highly parallelizable.

Original: However, the micro-patch generation of COCO-GAN uses padding in all feature maps while applying convolution.
Revised: However, the micro-patch generation of COCO-GAN uses padding in all feature maps while applying convolution.

Original: This problem causes a large number number of FLOPs for each image generation.
Revised: This problem causes a large number number of FLOPs for each image generation.

Original: We are particularly interested in this phenomenon and ﬂag utilizing the spatial disentanglement to reduce the total number of FLOPs as an important future work.
Revised: We are particularly interested in this phenomenon and ﬂag utilizing the spatial disentanglement to reduce the total number of FLOPs as an important future work.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this work, we propose  a new generative model, COCO-GAN.
Revised: In this work, we propose COCO-GAN , a new generative model toward dividing full image generation into non-overlapping patches generation.

Original: Through the experiments, we show that COCO-GAN can learn spatial disentanglement with limited partial views.
Revised: Through the experiments, we show that COCO-GAN can learn and inference with limited partial views.

Original: Although COCO-GAN is restrained from accessing the full scene, COCO-GAN can still generate high-quality samples without extra hyper-parameter tuning.
Revised: Although the model is restrained from accessing the full scene, it can still generate high-quality samples without extra hyper-parameter tuning.

Original: We also demonstrate COCO-GAN is a coordinatesystem-aware learning framework, and take panorama generation within a cylindrical coordinate system as case study.
Revised: We also demonstrate COCO-GAN is a coordinate-system-aware framework, and take panorama generation within a cylindrical coordinate system as case study.

Original: Furthermore, we highlight the advantages of COCO-GAN by showcasing three applications, including “Patch-Inspired image generation”, “User-Viewport-Aware Partial Scene Generation”, and “Computation-Friendly Generation”.
Revised: Furthermore, we highlight the advantages of COCO-GAN by showcasing three applications, including “Patch-Inspired image generation”, “User-Viewport-Aware Partial Scene Generation”, and “Computation-Friendly Generation”.

Original: Despite the generation quality of COCO-GAN being competitive with other state-of-the-art GANs without any post-processing, sometimes we still observe that local structures of generated samples may be discontinued or mottled.
Revised: Despite the generation quality of COCO-GAN being competitive with other state-of-the-art GANs without any post-processing, sometimes we still observe that local structures of generated samples may be discontinued or mottled.

Original: This indicates that extra reﬁnements and blending methods are still important for COCO-GAN to generate more stable and reliable samples.
Revised: This indicates that extra reﬁnements and blending methods are still important for COCO-GAN to generate more stable and reliable samples.

Original: We adopt a discrete uniform sampling strategy over spatial positions since we observe a huge drop in generation quality if we use continuous uniform sampling.
Revised: We adopt a discrete uniform sampling strategy over spatial positions since we observe a huge drop in generation quality with continuous uniform sampling.

Original: Although in practice COCO-GAN successfully learnsthe spatial disentanglement using discrete sampling, continuous sampling, in theory, should still be preferredover discrete sampling since the spatial domain is continuous.
Revised: Although in practice COCO-GAN successfully learns spatial continuity using discrete sampling, continuous sampling, in theory, should still be preferred since the spatial domain is continuous.

Original: Achieving such a goal would require deeper understanding and insights about the root cause of the generation quality drop.
Revised: Achieving such a goal would require deeper understanding and insights about the root cause of the generation quality drop.

Original: We demonstrate that COCO-GAN can generate panoramas under a cylindrical coordinate system.
Revised: We demonstrate that COCO-GAN can generate panoramas under a cylindrical coordinate system.

Original: However, another commonly used panorama format is sky-sphere under a hyperbolic coordinate system.
Revised: However, another commonly used panorama format is sky-sphere under a hyperbolic coordinate system.

Original: Consider that the image patches with the hyperbolic coordinate is not square-shaped, further studies on incorporating special convolution schemes like Spherical-CNN (Cohen et al., 2018) and implementing COCO-GAN under a hyperbolic coordinate system would be required.
Revised: Considering that the image patches with the hyperbolic coordinate is not square-shaped, further studies on incorporating special convolution schemes like Spherical-CNN (Cohen et al., 2018) and implementing COCO-GAN under a hyperbolic coordinate system would be required.

Revised: Furthermore, to allow a more ﬂexible and general coordinate system, some learnable coordinating methods (Balakrishnan et al., 2018) might be correlated to COCO-GAN and could further enhance the ﬂexibility.

Original: The size of micro patches is crucial to the results of COCO-GAN.
Revised: The size of micro patches is crucial to the results of COCO-GAN.

Original: A rule-of-thumb is that the size should be large enough to cover sufﬁcient information.
Revised: A rule-of-thumb is that the size should be large enough to cover sufﬁcient information.

Original: The precise lower bound requires experiments to examine if COCO-GAN learns undesired spatial patterns.
Revised: The precise lower bound requires experiments to examine if COCO-GAN learns undesired spatial patterns.

Original: Figure 5 shows that the model can be misled to learn reasonable but incorrect spatial relationship.
Revised: In “Spatial Positions Interpolations” of Section 3.2, we mention the model can be misled to learn reasonable but incorrect spatial relationship.

Original: An effective evaluation for the lower bound of themicro patch size is needed as future investigation.
Revised: An effective evaluation for the lower bound of the patch size needs future investigation.

Original: A PPENDIX A COCO-GAN DURING T ESTING P HASE
Revised: A PPENDIX A COCO-GAN DURING T ESTING P HASE

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX B
Revised: A [Equation] B

Original: M ODEL
Revised: M ODEL

Original: A RCHITECTURE D ETAILS
Revised: A RCHITECTURE D ETAILS

Original: [Figure]
Revised: [Figure]

Original: Generator 32x
Revised: Generator 32x

Original: [Figure]
Revised: [Figure]

Original: Generator Residual Block
Revised: Generator Residual Block

Original: [Figure]
Revised: [Figure]

Original: a size of 32 × 32 pixels.
Revised: a size of 32 × 32 pixels.

Original: We directly duplicate/remove the last residual block if we need to enlarge/reduce the size of output patch.
Revised: We directly duplicate/remove the last residual block if we need to enlarge/reduce the size of output patch.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Discriminator Residual Block
Revised: Discriminator Residual Block

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Input shape:
Revised: Input shape:

Original: [Table]
Revised: [Table]

Original: B, 512)
Revised: B, 512)

Original: [Table]
Revised: [Table]

Original: shape :
Revised: shape :

Original: [Table]
Revised: [Table]

Original: B, Hx
Revised: B, Hx

Original: [Table]
Revised: [Table]

Original: , D
Revised: , D

Original: [Table]
Revised: [Table]

Original: Output shape:
Revised: Output shape:

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: Discriminator Auxiliary Head
Revised: Discriminator Auxiliary Head

Original: Batch Normalization
Revised: Batch Normalization

Original: Linear Output shape: (B, 128)
Revised: Linear Output shape: (B, 128)

Original: Batch Normalization
Revised: Batch Normalization

Original: Linear Output shape: (B, 1) tanh
Revised: Linear Output shape: (B, 1) tanh

Original: [Figure]
Revised: [Figure]

Original: A PPENDIXC M ORE F ULL -I MAGE G ENERATION E XAMPLES
Revised: A PPENDIX

Revised: C M ORE F ULL -I MAGE G ENERATION E XAMPLES

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: A PPENDIXD M ORE I NTERPOLATION E XAMPLES
Revised: A PPENDIX

Revised: [Figure] A BLATION S TUDY [Figure]

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX E P ATCH -I NSPIRED I MAGE G ENERATION
Revised: A PPENDIX G P ATCH -I NSPIRED I MAGE G ENERATION

Original: [Figure]
Revised: [Figure]

Original: Since

Original: [Figure]
Revised: [Figure]

Original: Since the information loss of the cropping process (from (a) to (b)) is critical, we do not expect (a) and (c) to be identical.
Revised: Since the information loss of the cropping process (from (a) to (b)) is critical, we do not expect (a) and (c) to be identical.

Original: Instead, (b) and (d) should be visually similar and (d) should be globally coherent.
Revised: Instead, (b) and (d) should be visually similar and (d) should be globally coherent.

Original: Note that the diversity and difﬁculty of LSUN (bedroom category) is higher than
Revised: Note that the diversity and difﬁculty of LSUN (bedroom category) is higher than

Original: CelebA. The Q network can only capture the structure, shape, and orientation of the room and the bed, but it fails to capture detailed texture of objects .
Revised: CelebA. The Q network can only capture the structure, shape, and orientation of the room and the bed, but it fails to capture detailed texture of objects .

Original: A PPENDIX F T RAINING I NDICATORS
Revised: A PPENDIX H T RAINING I NDICATORS

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX G S PATIAL P OSITIONS I NTERPOLATION
Revised: A PPENDIX I S PATIAL P OSITIONS I NTERPOLATION