Original: NOT - SO - BIG -GAN: G ENERATING H IGH -F IDELITY I MAGES ON S MALL C OMPUTE WITH W AVELET - BASED S UPER -R ESOLUTION
Revised: NOT - SO - BIG -GAN: G ENERATING H IGH -F IDELITY I MAGES ON S MALL C OMPUTE WITH W AVELET - BASED S UPER -R ESOLUTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: State-of-the-art models for high-resolution image generation, such as BigGAN and VQVAE-2, require an incredible amount of compute resources and/or time (512 TPU-v3 cores) to train, putting them out of reach for the larger research community.
Revised: State-of-the-art models for high-resolution image generation, such as BigGAN and VQVAE-2, require an incredible amount of compute resources and/or time (512 TPU-v3 cores) to train, putting them out of reach for the larger research community.

Original: On the other hand, GAN-based image super-resolution models, such as
Revised: On the other hand, GAN-based image super-resolution models, such as

Original: ESRGAN, can not only upscale images to high dimensions, but also are efﬁcient to train.
Revised: ESRGAN, can not only upscale images to high dimensions, but also are efﬁcient to train.

Original: In this paper, we present NOT - SO - BIG -GAN ( NSB -GAN), a simple yet cost-effective two-step training framework for deep generative models (DGMs) of high-dimensional natural images.
Revised: In this paper, we present NOT - SO - BIG -GAN ( NSB -GAN), a simple yet cost-effective two-step training framework for deep generative models (DGMs) of high-dimensional natural images.

Original: First, we generate images in low-frequency bands by training a sampler in the wavelet domain.
Revised: First, we generate images in low-frequency bands by training a sampler in the wavelet domain.

Original: Then, we super-resolve these images from the wavelet domain back to the pixel-space with our novel wavelet super-resolution decoder network.
Revised: Then, we super-resolve these images from the wavelet domain back to the pixel-space with our novel wavelet super-resolution decoder network.

Original: Wavelet-based down-sampling method preserves more structural information than pixel-based methods, leading to signiﬁcantly better generative quality of the low-resolution sampler (e.g., 64 × 64).
Revised: Wavelet-based down-sampling method preserves more structural information than pixel-based methods, leading to signiﬁcantly better generative quality of the low-resolution sampler (e.g., 64 × 64).

Original: Since the sampler and decoder can be trained in parallel and operate on much lower dimensional spaces than end-to-end models, the training cost is substantially reduced.
Revised: Since the sampler and decoder can be trained in parallel and operate on much lower dimensional spaces than end-to-end models, the training cost is substantially reduced.

Original: On ImageNet 512 × 512, our model achieves a Fr´echet Inception Distance (FID) of 10.59 – beating the baseline BigGAN model – at half the compute (
Revised: On ImageNet 512 × 512, our model achieves a Fr´echet Inception Distance (FID) of 10.59 – beating the baseline BigGAN model – at half the compute (

Original: TPU-v3 cores).
Revised: TPU-v3 cores).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative modeling of natural images has achieved great success in recent years (Kingma &
Revised: Generative modeling of natural images has achieved great success in recent years (Kingma &

Original: Welling, 2013; Goodfellow et al., 2014; Arjovsky et al., 2017; Menick & Kalchbrenner, 2019; Zhang et al., 2018a).
Revised: Welling, 2013; Goodfellow et al., 2014; Arjovsky et al., 2017; Menick & Kalchbrenner, 2019; Zhang et al., 2018a).

Original: Advancements in scalable computing and theoretical understanding of generative models (Miyato et al., 2018; Zhang et al., 2018a;
Revised: Advancements in scalable computing and theoretical understanding of generative models (Miyato et al., 2018; Zhang et al., 2018a;

Original: Gulrajani et al., 2017; Mescheder et al., 2018; 2017; Roth et al., 2017; Nowozin et al., 2016; Srivastava et al., 2017; 2020; Karras et al., 2020), have, for the ﬁrst time, enabled the state-of-the-art techniques to generate photo-realistic images in higher dimensions than ever before (Brock et al., 2018;
Revised: Gulrajani et al., 2017; Mescheder et al., 2018; 2017; Roth et al., 2017; Nowozin et al., 2016; Srivastava et al., 2017; 2020; Karras et al., 2020), have, for the ﬁrst time, enabled the state-of-the-art techniques to generate photo-realistic images in higher dimensions than ever before (Brock et al., 2018;

Original: Razavi et al., 2019; Karras et al., 2020).
Revised: Razavi et al., 2019; Karras et al., 2020).

Original: Yet, generating high-dimensional complex data, such as ImageNet, still remains challenging and extremely resource intensive.
Revised: Yet, generating high-dimensional complex data, such as ImageNet, still remains challenging and extremely resource intensive.

Original: At the forefront of high-resolution image generation is BigGAN (Brock et al., 2018), a generative adversarial network (GAN) (Goodfellow et al., 2014) that tackles the curse of dimensionality (CoD) head-on, using the latest in scalable GPU-computing.
Revised: At the forefront of high-resolution image generation is BigGAN (Brock et al., 2018), a generative adversarial network (GAN) (Goodfellow et al., 2014) that tackles the curse of dimensionality (CoD) head-on, using the latest in scalable GPU-computing.

Original: This allows for training BigGAN with large mini-batch sizes (e.g., 2048), which greatly helps to model highly diverse, large-scale datasets like ImageNet.
Revised: This allows for training BigGAN with large mini-batch sizes (e.g., 2048), which greatly helps to model highly diverse, large-scale datasets like ImageNet.

Original: But, BigGAN’s ability to scale to high-dimensional data comes at the cost of a hefty compute budget.
Revised: But, BigGAN’s ability to scale to high-dimensional data comes at the cost of a hefty compute budget.

Original: A standard BigGAN model at 256 × 256 resolution can require up to a month or more of training time on as many as eight Tesla V100 graphics processing units (GPUs).
Revised: A standard BigGAN model at 256 × 256 resolution can require up to a month or more of training time on as many as eight Tesla V100 graphics processing units (GPUs).

Original: This compute requirement raises the barrier to entry for using and improving upon these technologies as the wider research community may not have access to any specialized hardware (e.g., Tensor processing units (TPUs) (Jouppi et al., 2017).
Revised: This compute requirement raises the barrier to entry for using and improving upon these technologies as the wider research community may not have access to any specialized hardware (e.g., Tensor processing units (TPUs) (Jouppi et al., 2017).

Original: The environmental impact of training large-scale models can also be substantial as training BigGAN on 512 × 512 images with 512 TPU cores for two days reportedly used as much electricity as the average American household does in about six months (Schwab, 2018).
Revised: The environmental impact of training large-scale models can also be substantial as training BigGAN on 512 × 512 images with 512 TPU cores for two days reportedly used as much electricity as the average American household does in about six months (Schwab, 2018).

Original: Motivated by these problems, we present NOT - SO - BIG -GAN ( NSB -GAN), a small compute training alternative to BigGAN, for class-conditional modeling of high-resolution images.
Revised: Motivated by these problems, we present NOT - SO - BIG -GAN ( NSB -GAN), a small compute training alternative to BigGAN, for class-conditional modeling of high-resolution images.

Original: In end-to-end generative models of high-dimensional data, such as VQVAE-2 (Razavi et al., 2019) and Karras et al.
Revised: In end-to-end generative models of high-dimensional data, such as VQVAE-2 (Razavi et al., 2019) and Karras et al.

Original: (2017), the lower layers transform noise into low resolution images, which are subsequently upscaled i.e. super-resolved to higher dimensions in the higher layers.
Revised: (2017), the lower layers transform noise into low resolution images, which are subsequently upscaled i.e. super-resolved to higher dimensions in the higher layers.

Original: Based on this insight, in
Revised: Based on this insight, in

Original: NSB -GAN we propose to split the end-to-end generative model into two separate neural networks, a sampler and an up-scaling decoder that can be trained in parallel on much smaller dimensional spaces.
Revised: NSB -GAN we propose to split the end-to-end generative model into two separate neural networks, a sampler and an up-scaling decoder that can be trained in parallel on much smaller dimensional spaces.

Original: In turn, we drastically reduce the compute budget of training.
Revised: In turn, we drastically reduce the compute budget of training.

Original: This split allows the sampler to be trained in up to 16-times lower dimensional space, not only making it compute efﬁcient, but also alleviating the training instability of end-to-end approaches.
Revised: This split allows the sampler to be trained in up to 16-times lower dimensional space, not only making it compute efﬁcient, but also alleviating the training instability of end-to-end approaches.

Original: To this end, we propose waveletspace training of GANs.
Revised: To this end, we propose waveletspace training of GANs.

Original: As compared to pixel-based interpolation methods for down-sampling images, wavelet-transform (WT) (Haar, 1909; Daubechies, 1992; Antonini et al., 1992) based downsampling preserves much more structural information, leading to much better samplers in fairly low resolutions (Sekar et al., 2014).
Revised: As compared to pixel-based interpolation methods for down-sampling images, wavelet-transform (WT) (Haar, 1909; Daubechies, 1992; Antonini et al., 1992) based downsampling preserves much more structural information, leading to much better samplers in fairly low resolutions (Sekar et al., 2014).

Original: When applied to a 2D image, wavelet transform slices the image into four equally-sized image-like patches along different frequency bands.
Revised: When applied to a 2D image, wavelet transform slices the image into four equally-sized image-like patches along different frequency bands.

Original: This process can be recursively applied multiple times in order to slice a large image into multiple smaller images, each representing the entire image in different bandwidths.
Revised: This process can be recursively applied multiple times in order to slice a large image into multiple smaller images, each representing the entire image in different bandwidths.

Original: This is diagrammatically shown in Figure 1.
Revised: This is diagrammatically shown in Figure 1.

Original: Here, the top-left patch (TL) lies in the lowest frequency band and contains most of the structure of the original image and therefore the only patch preserved during downsampling.
Revised: Here, the top-left patch (TL) lies in the lowest frequency band and contains most of the structure of the original image and therefore the only patch preserved during downsampling.

Original: The highly sparse top-right (TR), bottom-left (BL) and bottom-right (BL) patches lie in higher bands of frequency and are therefore dropped.
Revised: The highly sparse top-right (TR), bottom-left (BL) and bottom-right (BL) patches lie in higher bands of frequency and are therefore dropped.

Original: But wavelet-space sampling prohibits the use of existing pixel-space superresolution models, such as Ledig et al.
Revised: But wavelet-space sampling prohibits the use of existing pixel-space superresolution models, such as Ledig et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2018), to upscale the samples.
Revised: (2018), to upscale the samples.

Original: Thus, we introduce two wavelet-space super-resolution decoder networks that can work directly with waveletspace image encoding, while matching the performance of equivalent pixel-space methods.
Revised: Thus, we introduce two wavelet-space super-resolution decoder networks that can work directly with waveletspace image encoding, while matching the performance of equivalent pixel-space methods.

Original: Training our decoders is extremely compute efﬁcient (e.g., 3 days on the full ImageNet dataset), and, once trained on a diverse dataset like ImageNet, can generalize beyond the original training resolution and dataset.
Revised: Training our decoders is extremely compute efﬁcient (e.g., 3 days on the full ImageNet dataset), and, once trained on a diverse dataset like ImageNet, can generalize beyond the original training resolution and dataset.

Original: Our main contributions are the following:
Revised: Our main contributions are the following:

Original: • We introduce a simple training framework NSB -GAN for large generative models and show that super-resolving low-resolution samplers reduces the cost of training by orders of magnitude without sacriﬁcing image quality and even outperforms the baseline BigGAN model at higher resolution (e.g., 512 × 512 ).
Revised: • We introduce a simple training framework NSB -GAN for large generative models and show that super-resolving low-resolution samplers reduces the cost of training by orders of magnitude without sacriﬁcing image quality and even outperforms the baseline BigGAN model at higher resolution (e.g., 512 × 512 ).

Original: • In addition to a simple pixel-based version of NSB -GAN, we introduce wavelet-space training of DGMs and demonstrate that it leads to better image quality compared to pixel-space NSB -GAN model on average.
Revised: • In addition to a simple pixel-based version of NSB -GAN, we introduce wavelet-space training of DGMs and demonstrate that it leads to better image quality compared to pixel-space NSB -GAN model on average.

Original: • While conceptually simple (especially in the pixel-space), training large models to work in tandem requires careful engineering.
Revised: • While conceptually simple (especially in the pixel-space), training large models to work in tandem requires careful engineering.

Original: Therefore, we make our entire NSB -GAN code-base publicly available to the wider research community.
Revised: Therefore, we make our entire NSB -GAN code-base publicly available to the wider research community.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Given a set X
Revised: Given a set X

Original: = { x i |∀ i ∈ { 1 , . . .
Revised: = { x i |∀ i ∈ { 1 , . . .

Original: N } , x i ∈ R D } of samples from the true data distribution p ( x ) , the task of deep generative modeling is to approximate this distribution using deep neural networks.
Revised: N } , x i ∈ R D } of samples from the true data distribution p ( x ) , the task of deep generative modeling is to approximate this distribution using deep neural networks.

Original: All generative models of natural images assume that p ( x ) is supported on a K -dimensional, smaller manifold of R D .
Revised: All generative models of natural images assume that p ( x ) is supported on a K -dimensional, smaller manifold of R D .

Original: As such, they model the data using a conditional distribution p θ ( x | z ) , where θ represents the model parameters and Z ∈ R K is the latent variable with the prior p ( z ) .
Revised: As such, they model the data using a conditional distribution p θ ( x | z ) , where θ represents the model parameters and Z ∈ R K is the latent variable with the prior p ( z ) .

Original: Z is marginalized to obtain the log-likelihood (LLH) log p θ ( x ) = log (cid:82) p ( z ) p θ ( x | z ) dz of X under the model.
Revised: Z is marginalized to obtain the log-likelihood (LLH) log p θ ( x ) = log (cid:82) p ( z ) p θ ( x | z ) dz of X under the model.

Original: Generative model are trained by either maximizing this LLH or its lower-bound.
Revised: Generative model are trained by either maximizing this LLH or its lower-bound.

Original: There are two types of deep generative models, explicit models like variational autoencoders (VAE) (Kingma & Welling, 2013) that have an explicit form for the LLH and implicit models such as GANs (Goodfellow et al., 2014) that do not have such a form.
Revised: There are two types of deep generative models, explicit models like variational autoencoders (VAE) (Kingma & Welling, 2013) that have an explicit form for the LLH and implicit models such as GANs (Goodfellow et al., 2014) that do not have such a form.

Original: VAEs use a decoder network D θ to parameterize the model distribution p θ ( x | z ) and an encoder network E φ to parameterize a variational distribution q φ ( z | x ) as an approximation to the true posterior under the model.
Revised: VAEs use a decoder network D θ to parameterize the model distribution p θ ( x | z ) and an encoder network E φ to parameterize a variational distribution q φ ( z | x ) as an approximation to the true posterior under the model.

Original: They can then be trained using the evidence lower-bound (ELBO), log p θ ( x ) ≥ − KL [ q φ ( z | x ) (cid:107) p ( z )]
Revised: They can then be trained using the evidence lower-bound (ELBO), log p θ ( x ) ≥ − KL [ q φ ( z | x ) (cid:107) p ( z )]

Original: + (cid:82) q φ ( z | x ) log p θ ( x | z ) dz.
Revised: + (cid:82) q φ ( z | x ) log p θ ( x | z ) dz.

Original: KL here refers to the Kullback–Leibler divergence between q φ ( z | x ) and p ( z ) .
Revised: KL here refers to the Kullback–Leibler divergence between q φ ( z | x ) and p ( z ) .

Original: Unlike VAEs, GANs do not make any assumptions about the functional form of the conditional distribution p θ ( x | z ) and, therefore, do not have a tractable likelihood function.
Revised: Unlike VAEs, GANs do not make any assumptions about the functional form of the conditional distribution p θ ( x | z ) and, therefore, do not have a tractable likelihood function.

Original: Instead, they directly attempt to minimize either a variational lowerbound to a chosen f -divergence
Revised: Instead, they directly attempt to minimize either a variational lowerbound to a chosen f -divergence

Original: (Nowozin et al., 2016;
Revised: (Nowozin et al., 2016;

Original: Srivastava et al., 2017) or integral probability metric (IPM) (Arjovsky et al., 2017; Li et al., 2017; Srivastava et al., 2020) between the model distribution p θ ( x ) and p ( x ) using binary classiﬁer based density ratio estimators.
Revised: Srivastava et al., 2017) or integral probability metric (IPM) (Arjovsky et al., 2017; Li et al., 2017; Srivastava et al., 2020) between the model distribution p θ ( x ) and p ( x ) using binary classiﬁer based density ratio estimators.

Original: Though the idea of multiscale modeling in GANs has been considered throughly in Denton et al.
Revised: Though the idea of multiscale modeling in GANs has been considered throughly in Denton et al.

Original: ; Karras et al.
Revised: ; Karras et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2019b)
Revised: (2019b)

Original: , our method is more closely related to VQVAE2 (Razavi et al., 2019) and the Subscale Pixel Network (SPN) (Menick & Kalchbrenner, 2019) models.
Revised: , our method is more closely related to VQVAE2 (Razavi et al., 2019) and the Subscale Pixel Network (SPN) (Menick & Kalchbrenner, 2019) models.

Original: VQVAE-2 is a hierarchical version of the VQVAE model (van den Oord et al., 2017) that unlike
Revised: VQVAE-2 is a hierarchical version of the VQVAE model (van den Oord et al., 2017) that unlike

Original: VAEs, does not impose a KL-based regularization on the latent space.
Revised: VAEs, does not impose a KL-based regularization on the latent space.

Original: Instead, VQVAE/VQVAE models use vector quantization as a strong functional prior.
Revised: Instead, VQVAE/VQVAE models use vector quantization as a strong functional prior.

Original: As such, sampling from the latent space is not straightforward and requires a large compute budget to train auto-regressive PixelSNAIL (Chen et al., 2017) models to estimate the density on the latent space to generate samples.
Revised: As such, sampling from the latent space is not straightforward and requires a large compute budget to train auto-regressive PixelSNAIL (Chen et al., 2017) models to estimate the density on the latent space to generate samples.

Original: (Menick & Kalchbrenner, 2019) shows that it is easier to generate high-resolution images by splitting the process into two steps.
Revised: (Menick & Kalchbrenner, 2019) shows that it is easier to generate high-resolution images by splitting the process into two steps.

Original: In step one, they slice the image into small patches and then learn to generate these patches using a patch-level autoregressive model.
Revised: In step one, they slice the image into small patches and then learn to generate these patches using a patch-level autoregressive model.

Original: This step allows for capturing the general structure of the image but misses the ﬁner details.
Revised: This step allows for capturing the general structure of the image but misses the ﬁner details.

Original: Therefore, in step two, they train another network that learns to ﬁll in the missing details.
Revised: Therefore, in step two, they train another network that learns to ﬁll in the missing details.

Original: The SPN model has a signiﬁcant shortcoming, however, as its patch-level autoregressive approach fails to capture long term dependencies (e.g. eyes on a face looking in different directions).
Revised: The SPN model has a signiﬁcant shortcoming, however, as its patch-level autoregressive approach fails to capture long term dependencies (e.g. eyes on a face looking in different directions).

Original: Similar to SPN’s multi-scale approach, NSB -GAN decouples structure learning from adding details.
Revised: Similar to SPN’s multi-scale approach, NSB -GAN decouples structure learning from adding details.

Original: Unlike SPN, however, it does not suffer from the long-term dependency problem.
Revised: Unlike SPN, however, it does not suffer from the long-term dependency problem.

Original: This is because NSB -GAN uses wavelet transformation to slice the image, which naturally decouples the structure (TL) from the details (TR, BL, BR) in the image.
Revised: This is because NSB -GAN uses wavelet transformation to slice the image, which naturally decouples the structure (TL) from the details (TR, BL, BR) in the image.

Original: We demonstrate this difference diagrammatically in the Appendix F. More importantly, wavelet transform alleviates the need for autoregressive modeling of the patches as each patch represents the entire image in different frequency bands.
Revised: We demonstrate this difference diagrammatically in the Appendix F. More importantly, wavelet transform alleviates the need for autoregressive modeling of the patches as each patch represents the entire image in different frequency bands.

Original: Image Super Resolution Since the pioneering work of SRCNN (Dong et al., 2014) to tackle the problem of single image super-resolution, recovering a high-resolution image from a low-resolution image, with a deep neural network, deep convolutional neural network approaches have demonstrated great success reconstructing realistic high-resolution images.
Revised: Image Super Resolution Since the pioneering work of SRCNN (Dong et al., 2014) to tackle the problem of single image super-resolution, recovering a high-resolution image from a low-resolution image, with a deep neural network, deep convolutional neural network approaches have demonstrated great success reconstructing realistic high-resolution images.

Original: ESRGAN (Wang et al., 2018) is the state-of-the-art GAN model that majorly builds on SRGAN, RDN, and EDSR (Ledig et al., 2017;
Revised: ESRGAN (Wang et al., 2018) is the state-of-the-art GAN model that majorly builds on SRGAN, RDN, and EDSR (Ledig et al., 2017;

Original: Zhang et al., 2018b; Lim et al., 2017).
Revised: Zhang et al., 2018b; Lim et al., 2017).

Original: Improving upon its predecessor SRGAN, it modiﬁes the SRGAN’s architecture from SRResNet to Residual-in-Residual Dense Block (RRDB) without batch normalization, ﬁnetunes the perceptual loss implemented with the VGG-19 model (Simonyan & Zisserman, 2015), and utilizes a relativistic GAN-like  adversarial loss to predict relative realness.
Revised: Improving upon its predecessor SRGAN, it modiﬁes the SRGAN’s architecture from SRResNet to Residual-in-Residual Dense Block (RRDB) without batch normalization, ﬁnetunes the perceptual loss implemented with the VGG-19 model (Simonyan & Zisserman, 2015), and utilizes a relativistic GAN-like (Jolicoeur-Martineau, 2019) adversarial loss to predict relative realness.

Original: First, only the generator is trained as a Peak Signal-to-Noise Ratio (PSNR)-oriented model with L loss.
Revised: First, only the generator is trained as a Peak Signal-to-Noise Ratio (PSNR)-oriented model with L1 loss.

Original: Then, the GAN is trained as a whole, initialized from this pre-training, with carefully balanced L1, perceptual, and adversarial losses.
Revised: Then, the GAN is trained as a whole, initialized from this pre-training, with carefully balanced L1, perceptual, and adversarial losses.

Original: With such, ESRGAN is able to reconstruct realistic high-frequency texture and details missing in the low-resolution images.
Revised: With such, ESRGAN is able to reconstruct realistic high-frequency texture and details missing in the low-resolution images.

Original:  M ETHOD
Revised:  M ETHOD

Original: NSB -GAN training introduces a fairly simple change to the original BigGAN model.
Revised: NSB -GAN training introduces a fairly simple change to the original BigGAN model.

Original: Instead of training the generator directly on the full dimensionality of the data, D = 256 (say, for ImageNet dataset), we train the generator in K = 64 .
Revised: Instead of training the generator directly on the full dimensionality of the data, D = 256 (say, for ImageNet dataset), we train the generator in K = 64 .

Original: Then, in order to upscale the 64 × 64 image back to 256 , we train an up-scaling decoder.
Revised: Then, in order to upscale the 64 × 64 image back to 256 , we train an up-scaling decoder.

Original: We now describe two speciﬁc instances of the NSB -GAN framework, NSB -GAN-W and NSB -GAN-P in detail.
Revised: We now describe two speciﬁc instances of the NSB -GAN framework, NSB -GAN-W and NSB -GAN-P in detail.

Original: 3. NSB -GAN-W
Revised: 3. NSB -GAN-W

Original: NSB -GAN-W is best described as an autoencoder that operates in the frequency domain instead of the usual pixel space.
Revised: NSB -GAN-W is best described as an autoencoder that operates in the frequency domain instead of the usual pixel space.

Original: It is comprised of a deterministic encoder, a learned decoder and learned prior network, i.e. sampler.
Revised: It is comprised of a deterministic encoder, a learned decoder and learned prior network, i.e. sampler.

Original: These three components form a full generative model that can produce high-quality, high-resolution samples with a week of training on commodity hardware.
Revised: These three components form a full generative model that can produce high-quality, high-resolution samples with a week of training on commodity hardware.

Original: 3.1. D ETERMINISTIC E NCODER .
Revised: 3.1. D ETERMINISTIC E NCODER .

Original: The NSB -GAN-W encoder is a deterministic function that recursively applies wavelet transform to the input image, retaining only the TL patches at each level .
Revised: The NSB -GAN-W encoder is a deterministic function that recursively applies wavelet transform to the input image, retaining only the TL patches at each level .

Original: Each TL patch is a quarter of the size of the previous TL patch, resulting in a pyramid-like stack of 2D patches after multiple transformations.
Revised: Each TL patch is a quarter of the size of the previous TL patch, resulting in a pyramid-like stack of 2D patches after multiple transformations.

Original: The last TL contains the lowest frequencies and, therefore, the most structural information about the image.
Revised: The last TL contains the lowest frequencies and, therefore, the most structural information about the image.

Original: We use it as a compressed, lossy representation of the input image in our encoding step.
Revised: We use it as a compressed, lossy representation of the input image in our encoding step.

Original: For a N 2 -dimensional image X , let us denote the matrix obtained after wavelet transform as W ( X ) , which is a N × N block matrix with the following structure,
Revised: For a N 2 -dimensional image X , let us denote the matrix obtained after wavelet transform as W ( X ) , which is a N × N block matrix with the following structure,

Original: [Equation]
Revised: [Equation]

Original: Here, with a slight abuse of notation, W 1 , 1 ( X ) represents the (cid:0) N 2 × N 2 (cid:1) -dimensional
Revised: Here, with a slight abuse of notation, W 1 , 1 ( X ) represents the (cid:0) N 2 × N 2 (cid:1) -dimensional

Original: TL patch of image X. NSB -GAN’s encoder E can then be deﬁned recursively as follows,
Revised: TL patch of image X. NSB -GAN’s encoder E can then be deﬁned recursively as follows,

Original: [Equation]
Revised: [Equation]

Original: where l is the number of wavelet transforms applied to the input image.
Revised: where l is the number of wavelet transforms applied to the input image.

Original: As can be seen, the size of the retained TL patch decreases exponentially with l .
Revised: As can be seen, the size of the retained TL patch decreases exponentially with l .

Original: 3.1. D ECODER
Revised: 3.1. D ECODER

Original: After wavelet transform, the original image can be deterministically recovered from the TL, TR,
Revised: After wavelet transform, the original image can be deterministically recovered from the TL, TR,

Original: BL, and BR patches using IWT.
Revised: BL, and BR patches using IWT.

Original: NSB -GAN-W, however, discards the high-frequency patches during its encoding step, rendering a fully deterministic decoding impossible.
Revised: NSB -GAN-W, however, discards the high-frequency patches during its encoding step, rendering a fully deterministic decoding impossible.

Original: Thus, the NSB -GAN decoder ﬁrst learns to recover the missing high-frequency patches (using a neural network) and then deterministically combines them using IWT to reconstruct the original input.
Revised: Thus, the NSB -GAN decoder ﬁrst learns to recover the missing high-frequency patches (using a neural network) and then deterministically combines them using IWT to reconstruct the original input.

Original: We now present two
Revised: We now present two

Original: NSB -GAN-W decoder designs based on ResNet (He et al., 2016; Ledig et al., 2017) and UNet (Ronneberger et al., 2015) architectures respectively.
Revised: NSB -GAN-W decoder designs based on ResNet (He et al., 2016; Ledig et al., 2017) and UNet (Ronneberger et al., 2015) architectures respectively.

Original: ESRGAN-W Since NSB -GAN-W sampler generates samples in the wavelet domain, we modify the basic ESRGAN architecture (Wang et al., 2018) to allow it to accept images in the waveletdomain, up-scale them, and then project them back to the pixel-space.
Revised: ESRGAN-W Since NSB -GAN-W sampler generates samples in the wavelet domain, we modify the basic ESRGAN architecture (Wang et al., 2018) to allow it to accept images in the waveletdomain, up-scale them, and then project them back to the pixel-space.

Original: Speciﬁcally, we replace the generator in ESRGAN with a 16-block-deep SRResNet without batch normalization (Ledig et al., 2017), denoted by f θ , and replaced the bilinear interpolation based up-scaling with IWT.
Revised: Speciﬁcally, we replace the generator in ESRGAN with a 16-block-deep SRResNet without batch normalization (Ledig et al., 2017), denoted by f θ , and replaced the bilinear interpolation based up-scaling with IWT.

Original: We can then deﬁne the decoder as,
Revised: We can then deﬁne the decoder as,

Original: [Equation]
Revised: [Equation]

Original: This design not only allows us to directly input wavelet-encoded images in the decoder, but also lets us use the carefully balanced GAN-based training of ESRGAN since our decoder still operates in the pixel space to recover the missing higher frequencies.
Revised: This design not only allows us to directly input wavelet-encoded images in the decoder, but also lets us use the carefully balanced GAN-based training of ESRGAN since our decoder still operates in the pixel space to recover the missing higher frequencies.

Original: Similarly to ESRGAN, we pre-train
Revised: Similarly to ESRGAN, we pre-train

Original: SRResNet with only L1 Loss, initialize the generator at this state, and then start the full adversarial training with perceptual loss and adversarial loss.
Revised: SRResNet with only L1 Loss, initialize the generator at this state, and then start the full adversarial training with perceptual loss and adversarial loss.

Original: We refer to these two models as ResNet-W and
Revised: We refer to these two models as ResNet-W and

Original: ESRGAN-W , respectively.
Revised: ESRGAN-W , respectively.

Original: We defer further training details of the decoder to the Appendix C.2.
Revised: We defer further training details of the decoder to the Appendix C.2.

Original: UNet-Decoder While the ESRGAN-W decoder works well with wavelet-encoded inputs, it does not take full advantage of the compression that wavelet space modeling brings about.
Revised: UNet-Decoder While the ESRGAN-W decoder works well with wavelet-encoded inputs, it does not take full advantage of the compression that wavelet space modeling brings about.

Original: Therefore, we also introduce a UNet-based decoder that takes full advantage of the wavelet decomposition of 2D data to completely bypass the original dimensionality of the target image.
Revised: Therefore, we also introduce a UNet-based decoder that takes full advantage of the wavelet decomposition of 2D data to completely bypass the original dimensionality of the target image.

Original: Due to space limitations, we defer the details of this design to the Appendix C.3.
Revised: Due to space limitations, we defer the details of this design to the Appendix C.3.

Original: 3.1. S AMPLER : L EARNED P RIOR N ETWORK
Revised: 3.1. S AMPLER : L EARNED P RIOR N ETWORK

Original: The functional prior imposed by our deterministic encoder leads to a highly structured representation space made up of low frequency TL patches of images.
Revised: The functional prior imposed by our deterministic encoder leads to a highly structured representation space made up of low frequency TL patches of images.

Original: In order to generate from NSBGAN-W, one must be able to draw samples from this space.
Revised: In order to generate from NSBGAN-W, one must be able to draw samples from this space.

Original: We posit that, compared to sampling from equivalently-sized representation spaces for AE and VAEs, it is easier to sample from a lowdimensional, image-like latent space using generative methods such as GANs, as they repeatedly have been shown to excel in learning to samples from image distributions.
Revised: We posit that, compared to sampling from equivalently-sized representation spaces for AE and VAEs, it is easier to sample from a lowdimensional, image-like latent space using generative methods such as GANs, as they repeatedly have been shown to excel in learning to samples from image distributions.

Original: Therefore, we train a
Revised: Therefore, we train a

Original: BigGAN sampler on this representation space.
Revised: BigGAN sampler on this representation space.

Original: As the dimensionality is considerably lower than the original image, it only takes a single week and two Tesla V100 GPUs to to train the sampler.
Revised: As the dimensionality is considerably lower than the original image, it only takes a single week and two Tesla V100 GPUs to to train the sampler.

Original: Since the values of the TL patch may not lie in the [0 , 1] , range, care needs to be taken in order to normalize the encoded samplers properly as failing to do so prevents the BigGAN sampler to learn the correct distribution.
Revised: Since the values of the TL patch may not lie in the [0 , 1] , range, care needs to be taken in order to normalize the encoded samplers properly as failing to do so prevents the BigGAN sampler to learn the correct distribution.

Original: Pre-trained Sampler.
Revised: Pre-trained Sampler.

Original: It is possible to use a pre-trained BigGAN model at a lower resolution within NSB -GAN-W to generate higher resolution images.
Revised: It is possible to use a pre-trained BigGAN model at a lower resolution within NSB -GAN-W to generate higher resolution images.

Original: Consider a pre-trained BigGAN model that can generate 128 × 128 samples in the pixel-space.
Revised: Consider a pre-trained BigGAN model that can generate 128 × 128 samples in the pixel-space.

Original: By simply projecting the samples into the frequency-space using our deterministic encoder (down to 64 × 64) and then up-scaling it through our decoder network, one can generate samples at 256 × 256 without any signiﬁcant loss in quality.
Revised: By simply projecting the samples into the frequency-space using our deterministic encoder (down to 64 × 64) and then up-scaling it through our decoder network, one can generate samples at 256 × 256 without any signiﬁcant loss in quality.

Original: In fact, these samplers can outperform the end-to-end baseline BigGAN model not only in compute requirement, but also in terms of the FID at 512 × 512 resolution.
Revised: In fact, these samplers can outperform the end-to-end baseline BigGAN model not only in compute requirement, but also in terms of the FID at 512 × 512 resolution.

Original: 3. NSB -GAN-P
Revised: 3. NSB -GAN-P

Original: NSB -GAN-P is the pixel counterpart of NSB -GAN-W. Instead of encoding, generating, and decoding in the wavelet space, it operates in the pixel space.
Revised: NSB -GAN-P is the pixel counterpart of NSB -GAN-W. Instead of encoding, generating, and decoding in the wavelet space, it operates in the pixel space.

Original: The deterministic encoder sub-samples the original image in the pixel space to generate a low-resolution latent representation.
Revised: The deterministic encoder sub-samples the original image in the pixel space to generate a low-resolution latent representation.

Original: Speciﬁcally, we realize the sub-sampling as smoothing with a Gaussian ﬁlter and sub-sampling with respect to the upscaling factor r (Jo et al., 2018).
Revised: Speciﬁcally, we realize the sub-sampling as smoothing with a Gaussian ﬁlter and sub-sampling with respect to the upscaling factor r (Jo et al., 2018).

Original: For the NSB -GAN-P decoder, we use the ESRGAN model with the generator replaced with SRResNet without batch normalization, as in NSB -GAN-W, which we refer to as ESRGAN-P and the BigGAN model on low-dimensional space for the prior network.
Revised: For the NSB -GAN-P decoder, we use the ESRGAN model with the generator replaced with SRResNet without batch normalization, as in NSB -GAN-W, which we refer to as ESRGAN-P and the BigGAN model on low-dimensional space for the prior network.

Original: The training procedure is same as in ESRGAN-W, where ﬁrst we pre-train SRResNet with only L1 loss (namely, ResNet-P ) and then start adversarial training.
Revised: The training procedure is same as in ESRGAN-W, where ﬁrst we pre-train SRResNet with only L1 loss (namely, ResNet-P ) and then start adversarial training.

Original: Similar to the NSB -GAN-W model, we also have a UNet-version of the decoder but it reconstructs directly in the original data dimensionality like the ESRGAN based decoder.
Revised: Similar to the NSB -GAN-W model, we also have a UNet-version of the decoder but it reconstructs directly in the original data dimensionality like the ESRGAN based decoder.

Original: Further details are in Appendix C.2.
Revised: Further details are in Appendix C.2.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: As before, let X be the dataset of high-dimensional natural images.
Revised: As before, let X be the dataset of high-dimensional natural images.

Original: The NSB -GAN encoder can be deﬁned as a deterministic function E : R D (cid:55)→ R K that uses the WT to produce Z = { z j |∀ j ∈ { 1 , . . .
Revised: The NSB -GAN encoder can be deﬁned as a deterministic function E : R D (cid:55)→ R K that uses the WT to produce Z = { z j |∀ j ∈ { 1 , . . .

Original: N } , z j ∈ R K } .
Revised: N } , z j ∈ R K } .

Original: Using this paired dataset { X, Z } , we treat NSB -GAN as a fully observable model and train the decoder function D θ : R K (cid:55)→ R D to reconstruct X from Z by minimizing the negative log-likelihood (NLL) − E p ( x,z ) [log p θ ( x | z )] of the conditional probability distribution that it parameterizes.
Revised: Using this paired dataset { X, Z } , we treat NSB -GAN as a fully observable model and train the decoder function D θ : R K (cid:55)→ R D to reconstruct X from Z by minimizing the negative log-likelihood (NLL) − E p ( x,z ) [log p θ ( x | z )] of the conditional probability distribution that it parameterizes.

Original: Learning of the generator function G φ : R K (cid:55)→ R K which is referred to as learning the prior in previous literature (Razavi et al., 2019; De Fauw et al., 2019), is essentially ﬁtting a generative model to the marginal distribution of Z , i.e. p ( z ) .
Revised: Learning of the generator function G φ : R K (cid:55)→ R K which is referred to as learning the prior in previous literature (Razavi et al., 2019; De Fauw et al., 2019), is essentially ﬁtting a generative model to the marginal distribution of Z , i.e. p ( z ) .

Original: While we could use a similar approach as above and ﬁt G φ that parameterizes the model distribution p φ ( z ) by minimizing − E p ( z ) [log p φ ( z )] , we chose to instead use a BigGAN model to directly minimize the (variational lower-bound to the)
Revised: While we could use a similar approach as above and ﬁt G φ that parameterizes the model distribution p φ ( z ) by minimizing − E p ( z ) [log p φ ( z )] , we chose to instead use a BigGAN model to directly minimize the (variational lower-bound to the)

Original: f
Revised: f

Original: -divergence D f
Revised: -divergence D f

Original: [ p ( z ) (cid:107) p φ ( z )]
Revised: [ p ( z ) (cid:107) p φ ( z )]

Original: (Nowozin et al., 2016) between the true marginal distribution p ( z ) and the model distribution as it provides a better ﬁt for natural-image like distributions compared to other approaches.
Revised: (Nowozin et al., 2016) between the true marginal distribution p ( z ) and the model distribution as it provides a better ﬁt for natural-image like distributions compared to other approaches.

Original: All together, NSB -GAN simply speciﬁes a generative model over X and Z jointly and is, therefore, trained by minimising the NLL,
Revised: All together, NSB -GAN simply speciﬁes a generative model over X and Z jointly and is, therefore, trained by minimising the NLL,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ESRGAN
Revised: ESRGAN

Original: [Table]
Revised: [Table]

Original: P
Revised: P

Original: [Table]
Revised: [Table]

Original: ESRGAN
Revised: ESRGAN

Original: [Table] [Table]
Revised: [Table] [Table]

Original: None
Revised: None

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we quantitatively and qualitatively evaluate the training efﬁciency of the NSB -GAN approach against the end-to-end training approach of the BigGAN model by benchmarking the compute budget required vs. the quality of the generated samples.
Revised: In this section, we quantitatively and qualitatively evaluate the training efﬁciency of the NSB -GAN approach against the end-to-end training approach of the BigGAN model by benchmarking the compute budget required vs. the quality of the generated samples.

Original: While quantifying image quality remains a challenging problem (Borji, 2019), we report the Frechet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) as proxy measures to allow for comparison with BigGAN.
Revised: While quantifying image quality remains a challenging problem (Borji, 2019), we report the Frechet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) as proxy measures to allow for comparison with BigGAN.

Original: All results are reported on the ImageNet dataset (Deng et al., 2009) at two different resolutions
Revised: All results are reported on the ImageNet dataset (Deng et al., 2009) at two different resolutions

Original: 512 ×
Revised: 512 ×

Original: Compute Budget.
Revised: Compute Budget.

Original: For our main experiments on training efﬁciency, depending on the sampler used, we deﬁne two different compute budgets.
Revised: For our main experiments on training efﬁciency, depending on the sampler used, we deﬁne two different compute budgets.

Original: For the learned samplers, we train the NSB -GANW and NSB -GAN-P models for a total of 168 hours (7 full days).
Revised: For the learned samplers, we train the NSB -GANW and NSB -GAN-P models for a total of 168 hours (7 full days).

Original: Our training is performed on a single machine with four Telsa V100 GPUs with 16GB of VRAM each.
Revised: Our training is performed on a single machine with four Telsa V100 GPUs with 16GB of VRAM each.

Original: We found that training the baseline model (BigGAN) at 256x with only 4 GPUs in 7 days is not possible on the default setting, so we allow a total of 8 GPUs for the baseline training.
Revised: We found that training the baseline model (BigGAN) at 256x with only 4 GPUs in 7 days is not possible on the default setting, so we allow a total of 8 GPUs for the baseline training.

Original: Since the training of these BigGAN-based samplers are highly unstable (Zhao et al., 2020), we train 5 instances in total (3 BigGAN-deep and
Revised: Since the training of these BigGAN-based samplers are highly unstable (Zhao et al., 2020), we train 5 instances in total (3 BigGAN-deep and

Original: BigGAN samplers)
Revised: BigGAN samplers)

Original: and
Revised: and

Original: report results using the instance with the best FID score.
Revised: report results using the instance with the best FID score.

Original: For the pre-trained sampelrs, the compute budget is computed in terms of the TPUs used in their original training and does not include the decoder training.
Revised: For the pre-trained sampelrs, the compute budget is computed in terms of the TPUs used in their original training and does not include the decoder training.

Original: This is because the compute budget required to train the decoder is negligible compared to that of the sampler.
Revised: This is because the compute budget required to train the decoder is negligible compared to that of the sampler.

Original: Hyperparameters and Setup.
Revised: Hyperparameters and Setup.

Original: For the deterministic encoder using wavelet transformation, we instantiate it with biorthogonal 2.2 wavelet.
Revised: For the deterministic encoder using wavelet transformation, we instantiate it with biorthogonal 2.2 wavelet.

Original: For the NSB -GAN sampler, we use a batch size ofand learning rates of 10 − 4 and 4 × 10 4 for the generator and discriminator, respectively.
Revised: For the NSB -GAN sampler, we use a batch size ofand learning rates of 10 − 4 and 4 × 10 4 for the generator and discriminator, respectively.

Original: For the pretraining of ESRGAN-W and ESRGAN-P with L1 loss, we use a batch size of 32, and the learning rate is initialized 1 × 10 − 4 and decayed by a factor of 2 every 2 × 10 5 mini-batch updates.
Revised: For the pretraining of ESRGAN-W and ESRGAN-P with L1 loss, we use a batch size of 32, and the learning rate is initialized 1 × 10 − 4 and decayed by a factor of 2 every 2 × 10 5 mini-batch updates.

Original: During adversarial training of ESRGAN-W and ESRGAN-P, the learning rate is set to 1 × 10 − 4 and halved at [50 k, 100 k ] iterations.
Revised: During adversarial training of ESRGAN-W and ESRGAN-P, the learning rate is set to 1 × 10 − 4 and halved at [50 k, 100 k ] iterations.

Original: In the case of the UNet-decoders, for the ﬁrst-level decoder, we use a batch size of 128 and a learning rate of 10 − 4 .
Revised: In the case of the UNet-decoders, for the ﬁrst-level decoder, we use a batch size of 128 and a learning rate of 10 − 4 .

Original: For the second-level decoder, we use the same learning rate as the ﬁrst-level decoder, but a smaller batch size of 64 .
Revised: For the second-level decoder, we use the same learning rate as the ﬁrst-level decoder, but a smaller batch size of 64 .

Original: We allocate two GPUs to the NSB -GAN sampler, and the other two for the decoders.
Revised: We allocate two GPUs to the NSB -GAN sampler, and the other two for the decoders.

Original: For the UNet-decoder, this implies one GPU per level.
Revised: For the UNet-decoder, this implies one GPU per level.

Original: Each component is trained in parallel and independently from each other.
Revised: Each component is trained in parallel and independently from each other.

Original: To train the BigGAN baseline model at the native resolution of 256 × 256 , we use the same learning rates as for the NSBGAN sampler and a recommended batch size of 2048 (Brock et al., 2018).
Revised: To train the BigGAN baseline model at the native resolution of 256 × 256 , we use the same learning rates as for the NSBGAN sampler and a recommended batch size of 2048 (Brock et al., 2018).

Original: The baseline model trains with eight GPUs, instead of four like with NSB -GAN, to meet its large memory requirement, given the batch size and resolution of the image.
Revised: The baseline model trains with eight GPUs, instead of four like with NSB -GAN, to meet its large memory requirement, given the batch size and resolution of the image.

Original: We provide training and evaluation code for NSB -GAN sampler and the other models, respectively here: https://anonymous.4open.science/r/
Revised: We provide training and evaluation code for NSB -GAN sampler and the other models, respectively here: https://anonymous.4open.science/r/

Original: 4. R ESULTS 4.1. L EARNED S AMPLERS
Revised: 4. R ESULTS 4.1. L EARNED S AMPLERS

Original: To establish the training efﬁciency and competitive image quality of our NSB -GAN approach, we compare the FID and IS that the NSB -GAN-W, NSB -GAN-P and baseline BigGAN models reach in the compute budget of 7 days with four (8 for the baseline)
Revised: To establish the training efﬁciency and competitive image quality of our NSB -GAN approach, we compare the FID and IS that the NSB -GAN-W, NSB -GAN-P and baseline BigGAN models reach in the compute budget of 7 days with four (8 for the baseline)

Original: Tesla V100 GPUs in Table 1.
Revised: Tesla V100 GPUs in Table 1.

Original: Both NSBGAN-P and NSB -GAN-W models reach very competitive FIDs of 32.66 and 21.82, respectively.
Revised: Both NSBGAN-P and NSB -GAN-W models reach very competitive FIDs of 32.66 and 21.82, respectively.

Original: To put these results in perspective, even SAGAN (Zhang et al., 2018a), which requires twice the compute and operates on half the resolution (128 × 128), reaches an FID of ∼ 19.
Revised: To put these results in perspective, even SAGAN (Zhang et al., 2018a), which requires twice the compute and operates on half the resolution (128 × 128), reaches an FID of ∼ 19.

Original: Furthermore, in the same amount of time but twice the compute, the baseline BigGAN model fails to generate any meaningful samples across all ﬁve runs.
Revised: Furthermore, in the same amount of time but twice the compute, the baseline BigGAN model fails to generate any meaningful samples across all ﬁve runs.

Original: This illustrates that, compared to end-to-end models, the
Revised: This illustrates that, compared to end-to-end models, the

Original: NSB -GAN models are signiﬁcantly more compute efﬁcient and can reach competitive image quality.
Revised: NSB -GAN models are signiﬁcantly more compute efﬁcient and can reach competitive image quality.

Original: In the aforementioned models, we use the public PyTorch implementation of BigGAN (kindly provided by the authors) to provide fair comparisons between the NSB -GAN models and the baselines.
Revised: In the aforementioned models, we use the public PyTorch implementation of BigGAN (kindly provided by the authors) to provide fair comparisons between the NSB -GAN models and the baselines.

Original: It is important to note, however, that the Pytorch implementation is not optimal as it does not use
Revised: It is important to note, however, that the Pytorch implementation is not optimal as it does not use

Original: Sync-BatchNorm (SBN) which is essential for large mini-batch training.
Revised: Sync-BatchNorm (SBN) which is essential for large mini-batch training.

Original: This implementation has been reported to not reach the FID and IS reported in the original paper (ajbrock, 2018).
Revised: This implementation has been reported to not reach the FID and IS reported in the original paper (ajbrock, 2018).

Original: When the sampler is trained in the low-dimensional space of 64 × 64 , NSB -GAN-W clearly outperforms NSB -GAN-P.
Revised: When the sampler is trained in the low-dimensional space of 64 × 64 , NSB -GAN-W clearly outperforms NSB -GAN-P.

Original: We posit that this sharp difference in FID stems from the difference in the loss of structural information using the two down-sampling methods: pixel-based interpolation and wavelet-space encoding (Sekar et al., 2014).
Revised: We posit that this sharp difference in FID stems from the difference in the loss of structural information using the two down-sampling methods: pixel-based interpolation and wavelet-space encoding (Sekar et al., 2014).

Original: As illustrated in Figure 2 in Appendix A, WT encoding seems to preserve more structural information than pixel-based interpolation which misses key features of the image due to arbitrary sub-sampling.
Revised: As illustrated in Figure 2 in Appendix A, WT encoding seems to preserve more structural information than pixel-based interpolation which misses key features of the image due to arbitrary sub-sampling.

Original: This ﬁnding suggests that the down-sampled distribution at 64 × 64 becomes sufﬁciently different from the original distribution at 256 ×such that when trained on the down-sampled distribution, NSB -GAN-P sampler fails to sufﬁciently approximate the structure in the original data distribution.
Revised: This ﬁnding suggests that the down-sampled distribution at 64 × 64 becomes sufﬁciently different from the original distribution at 256 ×such that when trained on the down-sampled distribution, NSB -GAN-P sampler fails to sufﬁciently approximate the structure in the original data distribution.

Original: If this were true, the NSB -GAN-P performance should improve as the level of down-sampling is decreased.
Revised: If this were true, the NSB -GAN-P performance should improve as the level of down-sampling is decreased.

Original: With the following set of experiments with pre-trained samplers, we test this hypothesis and demonstrate that NSB -GAN framework can also be used with pre-trained samplers to make image generation at high resolution ( 256 × 256 , 512 × 512 ) efﬁcient and even beat the BigGAN baseline model in some cases.
Revised: With the following set of experiments with pre-trained samplers, we test this hypothesis and demonstrate that NSB -GAN framework can also be used with pre-trained samplers to make image generation at high resolution ( 256 × 256 , 512 × 512 ) efﬁcient and even beat the BigGAN baseline model in some cases.

Original: 4.1. P RE - TRAINED S AMPLERS
Revised: 4.1. P RE - TRAINED S AMPLERS

Original: We consider two pairs of pre-trained BigGAN samplers combined with our NSB -GAN models to ultimately generate at two different resolutions of 256 × 256 and 512 × 512.
Revised: We consider two pairs of pre-trained BigGAN samplers combined with our NSB -GAN models to ultimately generate at two different resolutions of 256 × 256 and 512 × 512.

Original: To clarify, our pretrained samplers are simply a serial combination of pre-trained BigGAN models 1 and our encoders.
Revised: To clarify, our pre-trained samplers are simply a serial combination of pre-trained BigGAN models 1 and our encoders.

Original: For example, Pretrained-128-64 is created by generating samples from a pre-trained sampler at 128 × 128 and then applying a down-sampling operation (pixel or wavelet) to obtain 64 × 64 samples.
Revised: For example, Pretrained-128-64 is created by generating samples from a pre-trained sampler at 128 ×and then applying a down-sampling operation (pixel or wavelet) to obtain 64 × 64 samples.

Original: Then, our decoders up-scale the low-resolution image four times to 256 × 256 .
Revised: Then, our decoders up-scale the low-resolution image four times to 256 × 256 .

Original: We report the FID/IS scores for all the samplers and compare to the baseline BigGAN models pre-trained at the respective resolutions in Table 2.
Revised: We report the FID/IS scores for all the samplers and compare to the baseline BigGAN models pre-trained at the respective resolutions in Table 2.

Original: First, notice that NSB -GAN-P performance increases drastically, supporting our hypothesis.
Revised: First, notice that NSB -GAN-P performance increases drastically, supporting our hypothesis.

Original: Next, both NSB -GAN-W and NSB -GAN-P models require signiﬁcantly less compute (up to 4 times less TPU-v3 cores) to train.
Revised: Next, both NSB -GAN-W and NSB -GAN-P models require signiﬁcantly less compute (up to 4 times less TPU-v3 cores, as shown in the last row of Table 2) to train.

Original: Most importantly, note how approach outperforms the original BigGAN model in terms of FID at 512 × 512 resolution, despite using exactly half the original compute budget.
Revised: Most importantly, note how approach outperforms the original BigGAN model in terms of FID at 512 × 512 resolution, despite using exactly half the original compute budget.

Original: We hypothesize that this is partly due to the way in which ImageNet 512 × 512 dataset is generated, but defer this discussion to Appendix H.
Revised: We hypothesize that this is partly due to the way in which ImageNet 512 × 512 dataset is generated, but defer this discussion to Appendix H.

Original: Furthermore, note that we did not have to re-train the decoders when applying to different resolutions and samplers.
Revised: Furthermore, note that we did not have to re-train the decoders when applying to different resolutions and samplers.

Original: All results in this experiment are done with the decoders from the previous experiment that were only trained to up-scale images from 64 × 64 to 256 × 256 .
Revised: All results in this experiment are done with the decoders from the previous experiment that were only trained to up-scale images from 64 × 64 to 256 × 256 .

Original: They evidently generalize well across different resolutions.
Revised: They evidently generalize well across different resolutions.

Original: This amortization of training across resolutions further reduces the training cost when more than one generator is trained.
Revised: This amortization of training across resolutions further reduces the training cost when more than one generator is trained.

Original: UNet-Decoders As shown in Table 4 of the Appendix B.1, our ESRGAN-based decoders clearly outperform our UNet-decoders on FID and IS.
Revised: UNet-Decoders As shown in Table 4 of the Appendix B.1, our ESRGAN-based decoders clearly outperform our UNet-decoders on FID and IS.

Original: It is important to note, however, that this difference is primarily because the ESRGAN decoders employ adversarial training which has been shown to drastically improve image quality as measured by FID.
Revised: It is important to note, however, that this difference is primarily because the ESRGAN decoders employ adversarial training which has been shown to drastically improve image quality as measured by FID.

Original: As shown in Table 3, not considering
Revised: As shown in Table 3, not considering

Original: [Table] [Table]
Revised: [Table] [Table]

Original: MSE on training and validation set for UNet-P, UNet-W,and VQVAE-2 models.
Revised: training and validation set for UNet-P, UNet-W,and VQVAE-2 models.

Original: Small difference between the training and validation error suggests that the models generalize well.
Revised: Small difference between the training and validation error suggests that the models generalize well.

Original: adversarial decoder, our UNet-W decoder leads to the best reconstruction error compared to the pixel-space decoder, including the state-of-the-art VQVAE-2 model.
Revised: adversarial decoder, our UNet-W decoder leads to the best reconstruction error compared to the pixel-space decoder, including the state-of-the-art VQVAE-2 model.

Original: Evaluation on LSUN Church with StyleGAN-2-W
Revised: Evaluation on LSUN Church with StyleGAN-2-W

Original: The NSB -GAN approach can be implemented with models other than BigGAN.
Revised: The NSB -GAN approach can be implemented with models other than BigGAN.

Original: To demonstrate this, we replace the BigGAN sampler architecture in our model with the StyleGAN-2 architecture and re-run the experiments on the LSUNChurch dataset.
Revised: To demonstrate this, we replace the BigGAN sampler architecture in our model with the StyleGAN-2 architecture and re-run the experiments on the LSUNChurch dataset.

Original: The results in Table 10 of the Appendix E demonstrate that our NSB -GAN models can generalize to different samplers and datasets.
Revised: The results in Table 10 of the Appendix E demonstrate that our NSB -GAN models can generalize to different samplers and datasets.

Original: Qualitative Results.
Revised: Qualitative Results.

Original: Due to space limitations, we have deferred all qualitative results to the Appendix.
Revised: Due to space limitations, we have deferred all qualitative results to the Appendix.

Original: Please see the Appendix A.1, A.2, A.3, and A.4 for a detailed qualitative comparison of all the models.
Revised: Please see the Appendix A.1, A.2, A.3, and A.4 for a detailed qualitative comparison of all the models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present a new genre of compute-efﬁcient generative models, NOT - SO - BIG -GAN, that achieve comparable image quality to the current SoTA DGM (BigGAN) with a dramatically lower compute budget.
Revised: In this work, we present a new genre of compute-efﬁcient generative models, NOT - SO - BIG -GAN, that achieve comparable image quality to the current SoTA DGM (BigGAN) with a dramatically lower compute budget.

Original: Surprisingly, NOT - SO - BIG -GAN is even able to outperform BigGAN in image quality at 512 × 512 resolution.
Revised: Surprisingly, NOT - SO - BIG -GAN is even able to outperform BigGAN in image quality at 512 × 512 resolution.

Original: Overall, we hope that our work inspires others to develop lowcompute generative models that can be utilized and iterated on by the wider research community.
Revised: Overall, we hope that our work inspires others to develop lowcompute generative models that can be utilized and iterated on by the wider research community.

Original: A A PPENDIX A. NSB -GAN S AMPLES WITH L EARNED S AMPLER
Revised: A A PPENDIX A. NSB -GAN S AMPLES WITH L EARNED S AMPLER

Original: Refer to Figure 3 for samples from NSB -GAN models
Revised: Refer to Figure 3 for samples from NSB -GAN models.

Original: . A. NSB -GAN S AMPLES WITH P RE - TRAINED S AMPLER
Revised: A. NSB -GAN S AMPLES WITH P RE - TRAINED S AMPLER

Original: Refer to Figures 4b and 4a for class-conditional samples from NSB -GAN-P and NSB -GAN-W models at 256 × 256 , respectively.
Revised: Refer to Figures 4b and 4a for class-conditional samples from NSB -GAN-P and NSB -GAN-W models at 256 × 256 , respectively.

Original: Refer to Figures 7a, 8a, 5a, and 6a for class-conditional samples from NSB -GAN-P and NSB -GAN-W models at 512 × 512 .
Revised: Refer to Figures 7a, 8a, 5a, and 6a for class-conditional samples from NSB -GAN-P and NSB -GAN-W models at 512 × 512 .

Original: Samples from BigGAN models at 256 × 256 and 512 × 512 are also included in 9, 10, and 11 for comparison.
Revised: Samples from BigGAN models at 256 × 256 and 512 × 512 are also included in 9, 10, and 11 for comparison.

Original: We intentionally include random samples from the same classes for all the models to allow for direct and easy comparison.
Revised: We intentionally include random samples from the same classes for all the models to allow for direct and easy comparison.

Original: A. F ULL R ESOLUTION S AMPLES FROM NSB -GAN-W WITH P RE - TRAINED S AMPLER
Revised: A. F ULL R ESOLUTION S AMPLES FROM NSB -GAN-W WITH P RE - TRAINED S AMPLER

Original: Refer to Figures 12, 13, 14, and 15 for full resolution samples from NSB -GAN-W at 256 × 256 .
Revised: Refer to Figures 12, 13, 14, and 15 for full resolution samples from NSB -GAN-W at 256 × 256 .

Original: Refer to Figures 16, 17,18, and 19 for 512 × 512 .
Revised: Refer to Figures 16, 17, and 19 for 512 × 512 .

Original: These samples are ﬁtted to the page.
Revised: These samples are ﬁtted to the page.

Original: [Table]
Revised: [Table]

Original: Refer to Figures 20, 21, and 23 for full resolution samples from NSB -GAN-W at 256 × 256 .
Revised: Refer to Figures 20, 21, and 23 for full resolution samples from NSB -GAN-W at 256 × 256 .

Original: Refer to Figures 24, 25, and 27 for 512 × 512 .
Revised: Refer to Figures 24, 25, and 27 for 512 × 512 .

Original: These samples are ﬁtted to the page.
Revised: These samples are ﬁtted to the page.

Original: B R ESULTS WITH T RUNCATION AND R EJECTION S AMPLING B. L EARNED S AMPLERS
Revised: B R ESULTS WITH T RUNCATION AND R EJECTION S AMPLING B. L EARNED S AMPLERS

Original: For a more thorough analysis of our NSB -GAN models with learned samplers, we apply truncation and rejection sampling to study their effects on FID and IS.
Revised: For a more thorough analysis of our NSB -GAN models with learned samplers, we apply truncation and rejection sampling to study their effects on FID and IS.

Original: We apply truncation at [1 . 0 , 0 . 8 , 0 . 6 , 0 . 4 , 0 . 2 , 0 . 1] and, given the truncation level at which the model outputs the best FID, we apply rejection sampling with a pre-trained Inception V3 model on ImageNet.
Revised: We apply truncation at [1 . 0 , 0 . 8 , 0 . 6 , 0 . 4 , 0 . 2 , 0 . 1] and, given the truncation level at which the model outputs the best FID, we apply rejection sampling with a pre-trained Inception V3 model on ImageNet.

Original: We test rejection sampling thresholds at 0.70, 0.80, 0.90, and 0.95.
Revised: We test rejection sampling thresholds at 0.70, 0.80, 0.90, and 0.95.

Original: Refer to Table 4 for quantitative results on FID and IS.
Revised: Refer to Table 4 for quantitative results on FID and IS.

Original: We also include FID and IS results for UNet-W decoder that was only trained with a MSE loss.
Revised: We also include FID and IS results for UNet-W decoder that was only trained with a MSE loss.

Original: [Figure]
Revised: [Figure]

Original: Sampler
Revised: Sampler

Original: Decoder
Revised: Decoder

Original: Resolution min FID / IS
Revised: Resolution min FID / IS

Original: FID / max IS
Revised: FID / max IS

Original: Pretrained
Revised: Pretrained

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ESRGAN-P 12.28 / 46.
Revised: ESRGAN-P 12.28 / 46.

Original: Pretrained
Revised: Pretrained

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ESRGAN-W 12.66 / 45.
Revised: ESRGAN-W 12.66 / 45.

Original: Pretrained
Revised: Pretrained

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ESRGAN-P 10.30 / 213.
Revised: ESRGAN-P 10.30 / 213.

Original: Pretrained-
Revised: Pretrained-

Original: [Table]
Revised: [Table]

Original: ESRGAN-W 10.59 / 52.
Revised: ESRGAN-W 10.59 / 52.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B. P RE - TRAINED S AMPLERS
Revised: B. P RE - TRAINED S AMPLERS

Original: We conduct the same detailed analysis for our NSB -GAN models with pre-trained samplers as the above.
Revised: We conduct the same detailed analysis for our NSB -GAN models with pre-trained samplers as the above.

Original: Refer to Table 5 for quantitative results on FID and IS.
Revised: Refer to Table 5 for quantitative results on FID and IS.

Original: [Figure]
Revised: [Figure]

Original: C A RCHITECTURE , H YPERPARAMETERS , AND T RAINING D ETAILS C. S AMPLER
Revised: C A RCHITECTURE , H YPERPARAMETERS , AND T RAINING D ETAILS C. S AMPLER

Original: Refer to 6 for details about the architecture and hyperparameters of the learned samplers (LearnedP-64 and Learned-W-64).
Revised: Refer to 6 for details about the architecture and hyperparameters of the learned samplers (LearnedP-64 and Learned-W-64).

Original: We attempted to train the sampler with both BigGAN and BigGAN-deep architectures and report on the models that achieved the best FID for each.
Revised: We attempted to train the sampler with both BigGAN and BigGAN-deep architectures and report on the models that achieved the best FID for each.

Original: Empirically, we found the models trained on the pixel-space much more unstable than on the wavelet-space, with four out of ﬁve models diverging.
Revised: Empirically, we found the models trained on the pixel-space much more unstable than on the wavelet-space, with four out of ﬁve models diverging.

Original: C. NSB -GAN D ECODERS
Revised: C. NSB -GAN D ECODERS

Original: ESRGAN-W and ESRGAN-P
Revised: ESRGAN-W and ESRGAN-P

Original: A similar training procedure in ESRGAN is conducted for both of our decoders, ESRGAN-W and ESRGAN-P. First, SRResNet, with batch normalization removed, is trained with L 1 loss:
Revised: A similar training procedure in ESRGAN is conducted for both of our decoders, ESRGAN-W and ESRGAN-P. First, SRResNet, with batch normalization removed, is trained with L 1 loss:

Original: L 1 = E x i
Revised: L 1 = E x i

Original: || G
Revised: || G

Original: ( x i )
Revised: ( x i )

Original: − y
Revised: − y

Original: || 1 , where x i is the down-sampled image and y is the target image.
Revised: || 1 , where x i is the down-sampled image and y is the target image.

Original: After training with this L 1 loss for 150k iterations, the GAN is trained with perceptual and adversarial losses added.
Revised: After training with this L 1 loss for 150k iterations, the GAN is trained with perceptual and adversarial losses added.

Original: Therefore, the total loss for the generator becomes:
Revised: Therefore, the total loss for the generator becomes:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Perceptual loss is implemented with a pre-trained VGG-19 model on ImageNet and we use the features of the fourth layer before the ﬁfth max-pooling layer.
Revised: Perceptual loss is implemented with a pre-trained VGG-19 model on ImageNet and we use the features of the fourth layer before the ﬁfth max-pooling layer.

Original: This GAN model is then trained for another 150k iterations.
Revised: This GAN model is then trained for another 150k iterations.

Original: As in the training for the learned sampler in wavelet domain, because the wavelet-encoded input image does not lie in the standard [0 , 1] range for ESRGAN-W, a normalization step is applied to transform the range into [0 , 1] .
Revised: As in the training for the learned sampler in wavelet domain, because the wavelet-encoded input image does not lie in the standard [0 , 1] range for ESRGAN-W, a normalization step is applied to transform the range into [0 , 1] .

Original: By using the minimum and maximum values of the waveletencoded input images, pre-calculated in the pre-processing step, we use the following normalization technique:
Revised: By using the minimum and maximum values of the waveletencoded input images, pre-calculated in the pre-processing step, we use the following normalization technique:

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: max x (cid:101) , where x i is the wavelet-encoded input and min x and max x are the pre-calculated minimum and maximum values of the wavelet-encoded input images.
Revised: max x (cid:101) , where x i is the wavelet-encoded input and min x and max x are the pre-calculated minimum and maximum values of the wavelet-encoded input images.

Original: Both ESRGAN-W and ESRGAN-P share the same architecture design, except for how the input is up-scaled and added to the learned features in order to induce the residual learning paradigm.
Revised: Both ESRGAN-W and ESRGAN-P share the same architecture design, except for how the input is up-scaled and added to the learned features in order to induce the residual learning paradigm.

Original: Refer to Table 7 for speciﬁc details of the architecture and hyperparameters.
Revised: Refer to Table 7 for speciﬁc details of the architecture and hyperparameters.

Original: C. UN ET
Revised: C. UN ET

Original: UNet in Wavelet Domain After a wavelet transform, the original image can be deterministically recovered from the TL, TR, BL, and BR patches using IWT.
Revised: UNet in Wavelet Domain After a wavelet transform, the original image can be deterministically recovered from the TL, TR, BL, and BR patches using IWT.

Original: NSB -GAN-W, however, discards the high-frequency patches (TR, BL, and BR) during its encoding step, rendering a fully deterministic
Revised: NSB -GAN-W, however, discards the high-frequency patches (TR, BL, and BR) during its encoding step, rendering a fully deterministic

Original: [Figure]
Revised: [Figure]

Original: decoding impossible.
Revised: decoding impossible.

Original: To resolve this, the NSB -GAN decoder ﬁrst learns to recover the missing high-frequency patches (using a neural network) and then deterministically combines them using
Revised: To resolve this, the NSB -GAN decoder ﬁrst learns to recover the missing high-frequency patches (using a neural network) and then deterministically combines them using

Original: IWT to reconstruct the original input.
Revised: IWT to reconstruct the original input.

Original: Since NSB -GAN’s encoding operation recursively applies wavelet transforms and discards the highfrequency components at each of the L encoding levels, we train L decoder networks to reconstruct the missing frequencies at each corresponding level of decoding.
Revised: Since NSB -GAN’s encoding operation recursively applies wavelet transforms and discards the highfrequency components at each of the L encoding levels, we train L decoder networks to reconstruct the missing frequencies at each corresponding level of decoding.

Original: To parallelize the training of these decoder networks, we perform multiple wavelet transforms to the original high-resolution dataset, generating L training sets of TL, TR, BL and BR patches.
Revised: To parallelize the training of these decoder networks, we perform multiple wavelet transforms to the original high-resolution dataset, generating L training sets of TL, TR, BL and BR patches.

Original: This allows us to independently train each of the decoder networks (in a supervised manner) to reconstruct the missing high-frequency patches conditioned on the corresponding TL.
Revised: This allows us to independently train each of the decoder networks (in a supervised manner) to reconstruct the missing high-frequency patches conditioned on the corresponding TL.

Original: This parallelization boosts the convergence rate of
Revised: This parallelization boosts the convergence rate of

Original: NSB -GAN, allowing for a fully trained decoder in under 48 hours.
Revised: NSB -GAN, allowing for a fully trained decoder in under 48 hours.

Original: Again with a slight abuse of notation, let W l 1 , 1 be the TL patch at level l and IWT =
Revised: Again with a slight abuse of notation, let W l 1 , 1 be the TL patch at level l and IWT =

Original: WT − 1 .
Revised: WT − 1 .

Original: We can write the decoder for level l as
Revised: We can write the decoder for level l as

Original: [Equation]
Revised: [Equation]

Original: for 1 ≤
Revised: for 1 ≤

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Here, each f l is a deep neural network that is trained to reconstruct one of the TR, BL, or BR patches at level l , conditioned on the TL patch ( W l 1 , 1 ).
Revised: Here, each f l is a deep neural network that is trained to reconstruct one of the TR, BL, or BR patches at level l , conditioned on the TL patch ( W l 1 , 1 ).

Original: The main challenge in high-resolution image generation lies in overcoming the curse of dimensionality.
Revised: The main challenge in high-resolution image generation lies in overcoming the curse of dimensionality.

Original: But by leveraging IWT, the original dimensionality of the image is completely bypassed in the NSB -GAN decoder.
Revised: But by leveraging IWT, the original dimensionality of the image is completely bypassed in the NSB -GAN decoder.

Original: Therefore, in practice, at each level, we further divide the TR, BL and BR patches by applying WT until we reach the patch dimensionality of 32x32.
Revised: Therefore, in practice, at each level, we further divide the TR, BL and BR patches by applying WT until we reach the patch dimensionality of 32x32.

Original: This can be done irrespective of the original dimensionality of the image.
Revised: This can be done irrespective of the original dimensionality of the image.

Original: We then reconstruct these patches in parallel and recover the patches and the original image by recursively applying IWT.
Revised: We then reconstruct these patches in parallel and recover the patches and the original image by recursively applying IWT.

Original: This ability of NSBGAN to bypass the original dimensionality of the input separates it from all other SR methods that operate in the pixel space.
Revised: This ability of NSBGAN to bypass the original dimensionality of the input separates it from all other SR methods that operate in the pixel space.

Original: Refer to Figure 28a and Table 8 for architecture details and hyperparameters of the UNet decoder for the wavelet-space.
Revised: Refer to Figure 28a and Table 8 for architecture details and hyperparameters of the UNet decoder for the wavelet-space.

Original: Refer to Figure 30 for the training schematic with the UNet decoders in the wavelet-space.
Revised: Refer to Figure 30 for the training schematic with the UNet decoders in the wavelet-space.

Original: Architecture We realize each of the L decoder neural networks f θ l with a slightly modiﬁed version of the UNet architecture (Ronneberger et al., 2015) rather than the commonly used transposedconvolution based architecture.
Revised: Architecture We realize each of the L decoder neural networks f θ l with a slightly modiﬁed version of the UNet architecture (Ronneberger et al., 2015) rather than the commonly used transposedconvolution based architecture.

Original: UNet is typically used for image segmentation and object detection tasks.
Revised: UNet is typically used for image segmentation and object detection tasks.

Original: As shown in Figure 28, UNet is an autoencoder architecture that has skip-connections from
Revised: As shown in Figure 28, UNet is an autoencoder architecture that has skip-connections from

Original: [Figure]
Revised: [Figure]

Original: each encoding layer to its corresponding decoding layer.
Revised: each encoding layer to its corresponding decoding layer.

Original: These skip connections copy and paste the encoding layer’s output into the decoding layer, allowing the decoder to exclusively focus on reconstructing only the missing, residual information.
Revised: These skip connections copy and paste the encoding layer’s output into the decoding layer, allowing the decoder to exclusively focus on reconstructing only the missing, residual information.

Original: This architectural design makes it a compelling ﬁt for decoding in NSB -GAN.
Revised: This architectural design makes it a compelling ﬁt for decoding in NSB -GAN.

Original: We modify the UNet architecture by appending three shallow networks to its output with each one reconstructing one of the three high-frequency patches.
Revised: We modify the UNet architecture by appending three shallow networks to its output with each one reconstructing one of the three high-frequency patches.

Original: This setup allows us to capture the dependencies between the high-frequency patches while also allowing sufﬁcient capacity to capture to their individual differences.
Revised: This setup allows us to capture the dependencies between the high-frequency patches while also allowing sufﬁcient capacity to capture to their individual differences.

Original: UNet in Pixel Domain
Revised: UNet in Pixel Domain

Original: The UNet-P decoder is also a partly-learned function that uses a modiﬁed UNet-based architecture.
Revised: The UNet-P decoder is also a partly-learned function that uses a modiﬁed UNet-based architecture.

Original: First, the encoded image is deterministically upsampled using an interpolation-based method.
Revised: First, the encoded image is deterministically upsampled using an interpolation-based method.

Original: This leads to a low-quality, blurry image at the same size as the original image.
Revised: This leads to a low-quality, blurry image at the same size as the original image.

Original: We then train a UNet to ﬁll in the missing details in a similar approach to image superresolution methods (Hu et al., 2019).
Revised: We then train a UNet to ﬁll in the missing details in a similar approach to image superresolution methods (Hu et al., 2019).

Original: Unlike the NSB -GAN decoder that circumvents the CoD by avoiding reconstructions in the original data dimensionality, the NSB -GAN-P decoder still has to operate on the full image size, resulting in a larger decoder.
Revised: Unlike the NSB -GAN decoder that circumvents the CoD by avoiding reconstructions in the original data dimensionality, the NSB -GAN-P decoder still has to operate on the full image size, resulting in a larger decoder.

Original: Refer to Figure 28b and Table 9 for architecture details and hyperparameters of the UNet decoder for the pixel-space.
Revised: Refer to Figure 28b and Table 9 for architecture details and hyperparameters of the UNet decoder for the pixel-space.

Original: D NSB -GAN VS NSB -GAN-P I NFORMATION C ONTENT
Revised: D NSB -GAN VS NSB -GAN-P I NFORMATION C ONTENT

Original: As we show in Figure 2, the information content in the latent embedding of NSB -GAN and NSBGAN-P are drastically different.
Revised: As we show in Figure 2, the information content in the latent embedding of NSB -GAN and NSBGAN-P are drastically different.

Original: The TL patches from NSB -GAN preserve more structural information in the image than the down-sampled image from NSB -GAN-P. It is evident in the ﬁgure that pixel-based down-sampling method misses key structures and features, such as the face of the person, structure of the boat, and structure of the accordion keyboard, whereas wavelet encoding does not.
Revised: The TL patches from NSB -GAN preserve more structural information in the image than the down-sampled image from NSB -GAN-P. It is evident in the ﬁgure that pixel-based down-sampling method misses key structures and features, such as the face of the person, structure of the boat, and structure of the accordion keyboard, whereas wavelet encoding does not.

Original: Learned-WLearned-PBest model type
Revised: Learned-WLearned-PBest model type

Original: BigGAN-deep
Revised: BigGAN-deep

Original: BigGAN
Revised: BigGAN

Original: Batch size
Revised: Batch size

Original: Learning rate of generator 1e1e-
Revised: Learning rate of generator 1e1e-

Original: Learning rate of discriminator 4e4e-
Revised: Learning rate of discriminator 4e4e-

Original: Attention resolution
Revised: Attention resolution

Original: Dimension of random noise ( z dim)
Revised: Dimension of random noise ( z dim)

Original: Number of resblocks per stage in generator/discriminator
Revised: Number of resblocks per stage in generator/discriminator

Original: [Table]
Revised: [Table]

Original: β
Revised: β

Original: Adam optimizer
Revised: Adam optimizer

Original: [Table]
Revised: [Table]

Original: Adam optimizer (cid:15) 1e1e-
Revised: Adam optimizer (cid:15) 1e1e-

Original: Training iterations
Revised: Training iterations

Original: [Table]
Revised: [Table]

Original: E E VALUATION ON LSUN C HURCH WITH S TYLE GAN- 
Revised: E E VALUATION ON LSUN C HURCH WITH S TYLE GAN-

Original: To demonstrate that the NSB -GAN paradigm can be different samplers and that our decoder trained on ImageNet generalizes fairly well to an unseen dataset, we conduct a set of experiments with the BigGAN sampler replaced with a StyleGAN-2 sampler.
Revised: To demonstrate that the NSB -GAN paradigm can be different samplers and that our decoder trained on ImageNet generalizes fairly well to an unseen dataset, we conduct a set of experiments with the BigGAN sampler replaced with a StyleGAN-2 sampler.

Original: We test the StyleGAN-2 sampler with two decoders: one trained on ImageNet and another trained on LSUN Church.
Revised: We test the StyleGAN-2 sampler with two decoders: one trained on ImageNet and another trained on LSUN Church.

Original: Illustrated in 10, as expected, the decoder that was only trained on the LSUN Church dataset outperforms our decoder trained on ImageNet.
Revised: Illustrated in 10, as expected, the decoder that was only trained on the LSUN Church dataset outperforms our decoder trained on ImageNet.

Original: However, even with our decoder, we reach a very competitive FID of 13.53, beating other recent two-step approaches, such as (Liu et al., 2019a).
Revised: However, even with our decoder, we reach a very competitive FID of 13.53, beating other recent two-step approaches, such as (Liu et al., 2019a).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Sampler
Revised: Sampler

Original: Decoder
Revised: Decoder

Original: Resolution
Revised: Resolution

Original: FID
Revised: FID

Original: IS
Revised: IS

Original: Pretrained
Revised: Pretrained

Original: [Table] [Table]
Revised: [Table] [Table]

Original: W
Revised: W

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Even our decoder trained only on ImageNet reaches a competitive FID of 13.53, but evidently can be improved by training a new decoder on LSUN Church dataset.
Revised: Even our decoder trained only on ImageNet reaches a competitive FID of 13.53, but evidently can be improved by training a new decoder on LSUN Church dataset.

Original: These results show that our NSBGAN models can work with different samplers than BigGAN and that our decoder generalizes fairly well to unseen dataset.
Revised: These results show that our NSBGAN models can work with different samplers than BigGAN and that our decoder generalizes fairly well to unseen dataset.

Original: Therefore, we do not go about proving the compute efﬁciency in this case.
Revised: Therefore, we do not go about proving the compute efﬁciency in this case.

Original: F S LICING
Revised: F S LICING

Original: Figure 32b, shows the difference between the slicing operations of SPN and NSB -GAN.
Revised: Figure 32b, shows the difference between the slicing operations of SPN and NSB -GAN.

Original: SPN slices the images across the spatial dimensions thus introducing long-term dependencies.
Revised: SPN slices the images across the spatial dimensions thus introducing long-term dependencies.

Original: In contrast, NSBGAN decomposes the image along different frequency bands that preserves the global structure of the image in each of the patches.
Revised: In contrast, NSBGAN decomposes the image along different frequency bands that preserves the global structure of the image in each of the patches.

Original: [Equation]
Revised: [Equation]

Original: Wavelet transformation of an image (illustrated in Figure 1) is a two-step recursive process that splits the original image into four equal-sized patches, each a representation of the entire image in different frequency bands.
Revised: Wavelet transformation of an image (illustrated in Figure 1) is a two-step recursive process that splits the original image into four equal-sized patches, each a representation of the entire image in different frequency bands.

Original: In the ﬁrst step, a low-pass ﬁlter (LPF) and a high pass-ﬁlter (HPF) are applied to the original image.
Revised: In the ﬁrst step, a low-pass ﬁlter (LPF) and a high pass-ﬁlter (HPF) are applied to the original image.

Original: This produces two patches of the same size as the original image.
Revised: This produces two patches of the same size as the original image.

Original: Since the application of LPF and HPF leads to redundancy, we can apply Shannon-Nyquist theorem to downsample these patches by half without losing any information.
Revised: Since the application of LPF and HPF leads to redundancy, we can apply Shannon-Nyquist theorem to downsample these patches by half without losing any information.

Original: In step two, the same process is repeated on the output of step one, splitting the original image into four equally-sized patches (TL, TR, BL and BR).
Revised: In step two, the same process is repeated on the output of step one, splitting the original image into four equally-sized patches (TL, TR, BL and BR).

Original: TR, BL and BR contain increasingly higher frequencies of the input image, preserving horizontal, vertical and diagonal edge information, respectively (contributing to the sharpness of the image).
Revised: TR, BL and BR contain increasingly higher frequencies of the input image, preserving horizontal, vertical and diagonal edge information, respectively (contributing to the sharpness of the image).

Original: H NSB -GAN AT 512 ×
Revised: H NSB -GAN AT 512 ×

Original: At 512 × 512 resolution, our model outperforms the BigGAN model.
Revised: At 512 × 512 resolution, our model outperforms the BigGAN model.

Original: We believe this to be the case mainly because of how the 512 × 512 training data is generated.
Revised: We believe this to be the case mainly because of how the 512 × 512 training data is generated.

Original: The ImageNet dataset is natively at 256 × 256 (approximately).
Revised: The ImageNet dataset is natively at 256 × 256 (approximately).

Original: When training the BigGAN model to generate 512 × 512, an interpolation-based method is used to upsample ImageNet images to 512 × 512, resulting in noise and blurriness in the upsampled images.
Revised: When training the BigGAN model to generate 512 × 512, an interpolation-based method is used to upsample ImageNet images to 512 × 512, resulting in noise and blurriness in the upsampled images.

Original: In comparison, our method takes samples at 128 × 128 and upsamples them using a learned SISR model.
Revised: In comparison, our method takes samples at 128 × 128 and upsamples them using a learned SISR model.

Original: This leads to substantially sharper images and therefore better FID scores.
Revised: This leads to substantially sharper images and therefore better FID scores.

Original: In fact, based on our study, it is better to use our model over an end-to-end
Revised: In fact, based on our study, it is better to use our model over an end-to-end

Original: DGM when learning to generate samples beyond the native resolution of the dataset.
Revised: DGM when learning to generate samples beyond the native resolution of the dataset.

Original: To clearly demonstrate this difference, we show samples from the bilinearly interpolated 

Revised: To clearly demonstrate this difference, we show samples from the bilinearly interpolated

Original: × 

Revised: ×

Original: ImageNet data and super-resolved version of the same samples with our decoder in Figure 31.
Revised: ImageNet data and super-resolved version of the same samples with our decoder in Figure 31.

Original: Clearly, interpolated images are blurrier than super-resolved images.
Revised: Clearly, interpolated images are blurrier than super-resolved images.

Original: Since BigGAN is trained to generate this blurry data, compared to our NSB -GAN approach, it performs sub-optimally.
Revised: Since BigGAN is trained to generate this blurry data, compared to our NSB -GAN approach, it performs sub-optimally.

Original: NSB -GAN beats the BigGAN baseline model on FID at 512 × 512 resolution.
Revised: NSB -GAN beats the BigGAN baseline model on FID at 512 × 512 resolution.

Original: We hypothesize that this difference in performance is due to the fact that ImageNet dataset at 512 × 512 is generated by using pixel-based interpolation up-sampling of the original data.
Revised: We hypothesize that this difference in performance is due to the fact that ImageNet dataset at 512 × 512 is generated by using pixel-based interpolation up-sampling of the original data.

Original: I O FFICIALLY P RE -T RAINED ESRGAN
Revised: I O FFICIALLY P RE -T RAINED ESRGAN

Original: Applying an ofﬁcially pre-trained ESRGAN does not perform as well as the ESRGAN-P andW models.
Revised: Applying an ofﬁcially pre-trained ESRGAN does not perform as well as the ESRGAN-P andW models.

Original: Speciﬁcally at 512 × 512 resolution, it suffers a slight increase in FID (10.73) and a substantial decrease ( > 100) in IS (52.25).
Revised: Speciﬁcally at 512 × 512 resolution, it suffers a slight increase in FID (10.73) and a substantial decrease ( > 100) in IS (52.25).

Original: J S AMPLING T IME
Revised: J S AMPLING T IME

Original: Sampling from NSB -GAN takes 0.039 s / image, whereas sampling from the BigGAN model takes 0.029 s / image, at 512 × 512 resolution.
Revised: Sampling from NSB -GAN takes 0.039 s / image, whereas sampling from the BigGAN model takes 0.029 s / image, at 512 × 512 resolution.

Original: Therefore, the overhead of upsampling (ESRGAN) is relatively small (0.010 s / image).
Revised: Therefore, the overhead of upsampling (ESRGAN) is relatively small (0.010 s / image).

Original: K F INETUNING NSB -GAN-W USING P RE - TRAINED B IG GAN
Revised: K F INETUNING NSB -GAN-W USING P RE - TRAINED B IG GAN

Original: Since the decoders are trained with real samples from the ImageNet dataset and tested on generated samples from our BigGAN models (learned and pre-trained), our up-sampled outputs could have visual artifacts due to the distribution mismatch.
Revised: Since the decoders are trained with real samples from the ImageNet dataset and tested on generated samples from our BigGAN models (learned and pre-trained), our up-sampled outputs could have visual artifacts due to the distribution mismatch.

Original: Therefore, we tried ﬁnetuning NSB -GAN-W endto-end with pre-trained BigGAN samples at 256 × 256 resolution.
Revised: Therefore, we tried ﬁnetuning NSB -GAN-W endto-end with pre-trained BigGAN samples at 256 × 256 resolution.

Original: We found end-to-end ﬁnetuning to increase high-frequency artifacts in the up-sampled images and showcase a few examples in Figure 32.
Revised: We found end-to-end ﬁnetuning to increase high-frequency artifacts in the up-sampled images and showcase a few examples in Figure 32.

Original: Finetuning also results in worse FID and IS scores of 13.73 and 44.02, respectively.
Revised: Finetuning also results in worse FID and IS scores of 13.73 and 44.02, respectively.

Original: Note that ﬁnetuning our model end-to-end requires ground truth samples from an already trained BigGAN at the target resolution.
Revised: Note that ﬁnetuning our model end-to-end requires ground truth samples from an already trained BigGAN at the target resolution.

Original: This somewhat defeats the purpose of our approach, which tries to avoid training a BigGAN at the target resolution to begin with.
Revised: This somewhat defeats the purpose of our approach, which tries to avoid training a BigGAN at the target resolution to begin with.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]