Original: MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
Revised: MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation

Original: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Although two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images, their quantization operator encodessimilar patches within an image into the same index, resulting in a repeated artifact for similar adjacent regions using existing decoder architectures.
Revised: Although two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images, their quantization operator encodes similar patches within an image into the same index, resulting in a repeated artifact for similar adjacent regions using existing decoder architectures.

Original: To addressthis issue, we propose to incorporate the spatially conditional normalization tomodulate the quantized vectors so as to insert spatially variant information to theembedded index maps, encouraging the decoder to generate more photorealisticimages.
Revised: To address this issue, we propose to incorporate the spatially conditional normalization to modulate the quantized vectors so as to insert spatially variant information to the embedded index maps, encouraging the decoder to generate more photorealistic images.

Original: Moreover, we use multichannel quantization to increase the recombinationcapability of the discrete codes without increasing the cost of model and codebook.
Revised: Moreover, we use multichannel quantization to increase the recombination capability of the discrete codes without increasing the cost of model and codebook.

Original: Additionally, to generate discrete tokens at the second stage, we adopt a Masked
Revised: Additionally, to generate discrete tokens at the second stage, we adopt a Masked

Original: Generative Image Transformer (MaskGIT) to learn an underlying prior distributionin the compressed latent space, which is much faster than the conventional autoregressive model.
Revised: Generative Image Transformer (MaskGIT) to learn an underlying prior distribution in the compressed latent space, which is much faster than the conventional autoregressive model.

Original: Experiments on two benchmark datasets demonstrate that ourproposed modulated VQGAN is able to greatly improve the reconstructed imagequality as well as provide high-fidelity image generation.
Revised: Experiments on two benchmark datasets demonstrate that our proposed modulated VQGAN is able to greatly improve the reconstructed image quality as well as provide high-fidelity image generation.

Original:  Introduction
Revised:  Introduction

Original: The vision community has rapidly improved image synthesis results on quality, diversity and resolution in the past few years.
Revised: The vision community has rapidly improved image synthesis results on quality, diversity and resolution over a short period of time.

Original: Many powerful baseline frameworks have been introduced, such as Generative Adversarial Networks (GAN) [13], Variational AutoEncoder (VAE) [23], Flow-based Generators [8] and Diffusion Probabilistic Models (DPM) [35].
Revised: In particular, many powerful baseline frameworks have been introduced, such as Generative Adversarial Networks (GAN) [13], Variational AutoEncoder (VAE) [23], Flow-based Generators [8] and Diffusion Probabilistic Models (DPM) [35].

Original: These methods are conceptually intuitive, leading to an explosion of image synthesis works that push the boundaries of image generationquality and diversity [14, 1, 22, 20, 21, 37, 4].
Revised: These methods are conceptually intuitive, leading to an explosion of image synthesis works that push the boundaries of image generation quality and diversity [14, 1, 22, 20, 21, 37, 4]

Original: Among the existing image synthesis frameworks, Vector Quantized-Variational AutoEncoder (VQVAE) [39] is a very popular nova, which learns a compressed discrete representation for each imageat the first stage and subsequently learns the underlying prior distribution in the discrete latent spaceat the second stage.
Revised: Among these existing image synthesis frameworks, Vector Quantized-Variational AutoEncoder (VQVAE) [39] is a very popular nova, which learns a compressed discrete representation for each image at the first stage and subsequently learns the underlying prior distribution in the discrete latent space at the second stage.

Original: Unlike conventional GANs that pursue a fragile balance in a minimax game,resulting in unstable training, the optimization target of VQ-VAE is a definite negative log-likelihood (NLL) objective of the training data.
Revised: Unlike conventional GANs that pursue a fragile balance in a minimax game, resulting in unstable training, the optimization target of VQ-VAE is a definite negative log-likelihood (NLL) objective of the training data.

Original: By maximizing the log-likelihood over all training examples,theoretically VQ-VAE is able to cover all modes of the data, and thus bypass the “mode collapse”issue in GAN.
Revised: By maximizing the log-likelihood over all training examples, theoretically VQ-VAE is able to cover all modes of the data, and thus bypass the “mode collapse” issue in GAN.

Original: On the other hand, compared with VAE [23], VQ-VAE maps an image into a palette oflatent discrete codes in higher resolution with spatial structure information and learns the compositionof the codes from the data itself, which overcomes the long-dragged image quality issue of VAE inimage synthesis.
Revised: On the other hand, compared with VAE [23], VQ-VAE maps an image into a palette of latent discrete codes in higher resolution with spatial structure information and learns the composition of the codes from the data itself, which overcomes the long-dragged image quality issue of VAE in image synthesis.

Original: These advantages of VQ-VAE have led to remarkable image synthesis results asevident by its recent extensions, such as VQ-VAE-2 [31], DALL-E [30], VQGAN [11], ImageBART[10], LDMs [32], VIT-VQGAN [45], RQ-VAE [24], MaskGIT [2] and DALL-E-2 [29].
Revised: These advantages of VQ-VAE have led to remarkable image synthesis results as evident by its recent extensions, such as VQ-VAE-2 [31], DALL-E [30], VQGAN [11], ImageBART [10], LDMs [32], VIT-VQGAN [45], RQ-VAE [24], MaskGIT [2] and DALL-E-2 [29].

Original: Despite the great performance, the VQ-VAE or VQGAN pipeline also has its shortcomings.
Revised: Despite the great performance, the VQ-VAE or VQGAN pipeline also has its shortcomings.

Original: Forexample, its second stage is typically modelled as a sequence generation process in an autoregressiveway [39, 11], i.e. generating each discrete latent code one by one at different spatial locations, whichis very time-consuming for inference.
Revised: For example, its second stage is typically modelled as a sequence generation process in an autoregressive way [39, 11], i.e. generating each discrete latent code one by one at different spatial locations, which is very time-consuming for inference.

Original: MaskGIT [2] nicely addresses this issue by predicting multipletokens based on the prediction confidence at each step, which greatly reduces the number of stepsneeded in the autoregressive token generator.
Revised: MaskGIT [2] nicely addresses this issue by predicting multiple tokens based on the prediction confidence at each step, which greatly reduces the number of steps needed in the autoregressive token generator.

Original: In contrast, in this paper we focus on improving the stage-1 learning of the VQ-based imagesynthesis pipeline.
Revised: In contrast, in this paper we focus on improving the stage-1 learning of the VQ-based image synthesis pipeline.

Original: Specifically, we notice another inherent shortcoming of the VQ-based methods,i.e. they often generate repeated artifact patterns in image synthesis (see Fig. 3), which is becausethe quantization operator drops away some variances and embeds similar patches into the samequantization index.
Revised: Specifically, we notice another inherent shortcoming of the VQ-based methods, i.e. they often generate repeated artifact patterns in image synthesis (see Fig. 3), which is because the quantization operator drops away some variances and embeds similar patches into the same quantization index.

Original: To address this, motivated by the spatially conditional normalization and positionembedding literature [17, 27, 20, 41], we re-design the decoder architecture in a way that utilizesa spatially conditional normalization layer to modulate the activations using the quantized vectors,which essentially adds spatially variant information to the discrete representation.
Revised: To address this, motivated by the spatially conditional normalization and position embedding literature [17, 27, 20, 41], we re-design the decoder architecture in a way that utilizes a spatially conditional normalization layer to modulate the activations using the quantized vectors, which essentially adds spatially variant information to the discrete representation.

Original: Moreover, weleverage a multichannel representation technique, introduced in [47] for image completion, to keepthe codebook size manageable while with sufficient representation capability for image generation.
Revised: Moreover, we leverage a multichannel representation technique, introduced in [47] for image completion, to keep the codebook size manageable while with sufficient representation capability for image generation.

Original: Lastly, with a better quantizer, we follow MaskGIT [2] to speed up the prior distribution training overthe compact discrete representations in the second stage [45].
Revised: Lastly, with a better quantizer, we follow MaskGIT [2] to speed up the prior distribution training over the compact discrete representations in the second stage [45].

Original: In summary, our main contributions are as follows:
Revised: In summary, our main contributions are as follows:

Original: • We point out an inherent shortcoming of the VQ-based image synthesis pipeline: having repeated artifacts in  similar nearby regions, and solve it by introducing a spatially conditional normalization to provide spatially variant information for different locations.
Revised: • We point out an inherent shortcoming of the VQ-based image synthesis pipeline: having repeated artifacts in semantically similar nearby regions, and solve it by introducing a spatially conditional normalization to provide spatially variant information for different locations.

Original: • With the help of our proposed spatially conditional normalization, we further incorporate the multichannel representation that dramatically improves the reconstructed image quality 
Revised: • With the help of our proposed spatially conditional normalization, we further incorporate the multichannel representation that dramatically improves the reconstructed image quality using the same encoder-decoder layers as in VQGAN.

Original: using the same encoder-decoder layers as in VQGAN.• When sampled from a learned prior on all directions for multi-locations and multi-channels,experimental results on two benchmark datasets show that our synthesized samples hold notonly high quality but also large diversity.
Revised: • When sampled from a learned prior on all directions for multi-locations and multi-channels, experimental results on two benchmark datasets show that our synthesized samples hold not only high quality but also large diversity.

Original:  Related Work
Revised:  Related Work

Original: VQ-based image synthesis.
Revised: VQ-based image synthesis.

Original: Our model is derived from VQGAN [11], which is built upon the twostage VQ-VAE [39]: first, a quantizer with an encoder-decoder architecture is trained to embed imagesinto compact sequences using discrete tokens from a learned codebook, and then a prior network islearned to model the underlying distribution in the discrete space.
Revised: Our model is derived from VQGAN [11], which is built upon the twostage VQ-VAE [39]: first, a quantizer with an encoder-decoder architecture is trained to embed images into compact sequences using discrete tokens from a learned codebook, and then a prior network is learned to model the underlying distribution in the discrete space.

Original: Unlike the minimax game in popular
Revised: Unlike the minimax game in popular

Original: GAN models [13, 1, 20, 21], the VQ-based generator is trained by optimizing negative log-likelihoodover all examples in the training set, leading to a stable training and bypassing the “mode collapse”issue.
Revised: GAN models [13, 1, 20, 21], the VQ-based generator is trained by optimizing negative log-likelihood over all examples in the training set, leading to a stable training and bypassing the “mode collapse” issue.

Original: Driven by these advantages, many image synthesis models follow the two-stage paradigm,such as image generation [31, 45, 2, 24, 16], image-to-image translation [11, 10, 32], text-to-imagesynthesis [30, 29, 10, 7], conditional video generation [28, 42, 44], and image completion [11, 10, 47].
Revised: Driven by these advantages, many image synthesis models follow the two-stage paradigm, such as image generation [31, 45, 2, 24, 16], image-to-image translation [11, 10, 32], text-to-image synthesis [30, 29, 10, 7], conditional video generation [28, 42, 44], and image completion [11, 10, 47].

Original: Apart from VQGAN, the most related works also include ViT-VQGAN [45] and RQ-VAE [24] thataim to train a better quantizer in the first stage.
Revised: Apart from VQGAN, the most related works also include ViT-VQGAN [45] and RQ-VAE [24] that aim to train a better quantizer in the first stage.

Original: Compared to them, our model is simple and efficient,yet effective to improve the image quality, without adding the computational cost on higher resolutionrepresentations [45] or more stages of recursive quantization [24].
Revised: Compared to them, our model is simple and efficient, yet effective to improve the image quality, without adding the computational cost on higher resolution representations with much larger model [45] or more stages of recursive quantization [24].

Original: Spatially conditional normalization, also called adaptive instance normalization, has been widelyused in image synthesis task [9, 17, 18, 27, 20, 41].
Revised: Spatially conditional normalization, also called adaptive instance normalization, has been widely used in image synthesis task [9, 17, 18, 27, 20, 41].

Original: Among them, the spatially conditional inputcontains various variants, such as style images [9, 17], target domain images [18], semantic maps [27], learned vectors [20] and random heatmaps [41].
Revised: Among them, the spatially conditional input contains various variants, such as style images [9, 17], target domain images [18], semantic maps [27], learned vectors [20] and random heatmaps [41].

Original: Comparing our approach to these works, ourspatially conditional input is the embedded discrete features, which contain  learned compact contents.
Revised: Comparing our approach to these works, our spatially conditional input is the embedded discrete features, which contain automatically learned compact contents.

Original: To the best of our knowledge, this is the first work to modulate quantized vectors and use multichannelquantization on the VQ-based image generation framework.
Revised: To the best of our knowledge, this is the first work to modulate quantized vectors and use multichannel quantization on the VQ-based image generation framework.

Original: In the following sections, we willdescribe the modulated quantized vectors and multichannel quantization and discuss their advantagesover the concurrent models such as [32, 45] and [24] in detail.[Figure] 𝑧𝑧 𝑞𝑞 [Figure]
Revised: In the following sections, we will describe the modulated quantized vectors and multichannel quantization and discuss their advantages over the concurrent models such as [32, 45] and [24] in detail.

Original: 𝐹𝐹 𝛽𝛽 ( ℎ , 𝑤𝑤 , 𝑐𝑐 ) ( ℎ

Original: [Figure]
Revised: [Figure]

Original: 𝑤𝑤

Original: [Figure]

Original: 𝑐𝑐 )

Original: [Figure]

Original: ( ℎ

Original: [Figure]

Original: 𝑤𝑤

Original: [Figure]

Original:  Method
Revised:  Method

Original: The proposed approach, illustrated in Fig.
Revised: The proposed approach, illustrated in Fig.

Original: 2, follows a two-stage recipe that embeds images usinga learnable codebook (see Fig.
Revised: 2, follows a two-stage recipe that embeds images using a learnable codebook (see Fig.

Original: 2 left), and then tames a bidirectional transformer to estimate theunderlying prior distribution over the discrete latent space (see Fig.
Revised: 2 left), and then tames a bidirectional transformer to estimate the underlying prior distribution over the discrete latent space (see Fig.

Original: 2 right).
Revised: 2 right).

Original: The key observationwe found is that a better quantizer will naturally lead to better image reconstruction and generationquality [45].
Revised: The key observation we found is that a better quantizer will naturally lead to better image reconstruction and generation quality [45].

Original: Hence, our main goal in this work is to improve the quantization in the first stage, yetwithout suffering from awkward computational cost as in concurrent works [45, 24].
Revised: Hence, our main goal in this work is to improve the quantization in the first stage, yet without suffering from awkward computational cost as in concurrent works [45, 24].

Original: 3.1 Modulating Quantized Vector
Revised: 3.1 Modulating Quantized Vector

Original: Background.
Revised: Background.

Original: Given an image x ∈ R H × W × 3 , vanilla VQ-VAEs learn a discrete codebook to represent observations as a collection of codebook entries z q ∈ R h × w × n q , where n q is the dimensionalityof quantized vectors in the codebook.
Revised: Given an image x ∈ R H × W × 3 , vanilla VQ-VAEs learn a discrete codebook to represent observations as a collection of codebook entries z q ∈ R h × w × n q , where n q is the dimensionality of quantized vectors in the codebook.

Original: In this way, each image can be equivalently represented asa compact sequence s with h · w indices of the codevectors z q .
Revised: In this way, each image can be equivalently represented as a compact sequence s with h · w indices of the codevectors z q .

Original: Formally, the observed image x isreconstructed by:
Revised: Formally, the observed image x is reconstructed by:

Original: [Equation]
Revised: [Equation]

Original: In particular, an encoder E ψ ( · ) first embeds an image x into a continuous vector ˆ z , and the quantizationoperator q ( · ) is then conducted to transfer the continuous feature ˆ z into the discrete space by lookingup the closest codebook entry z k for each spatial grid feature ˆ z ij within ˆ z :
Revised: In particular, an encoder E ψ ( · ) first embeds an image x into a continuous vector ˆ z , and the quantization operator q ( · ) is then conducted to transfer the continuous feature ˆ z into the discrete space by looking up the closest codebook entry z k for each spatial grid feature ˆ z ij within ˆ z :

Original: [Equation]
Revised: [Equation]

Original: where Z ∈ R K × n q is the codebook that consists of K entries with n q dimensions.
Revised: where Z ∈ R K × n q is the codebook that consists of K entries with n q dimensions.

Original: The quantizedvector z q is finally transmitted to a decoder G θ ( · ) for rebuilding the original image.
Revised: The quantized vector z q is finally transmitted to a decoder G θ ( · ) for rebuilding the original image.

Original: The overallmodels and the codebook can be learned by optimizing the following objective:
Revised: The overall models and the codebook can be learned by optimizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: Here, sg denotes the stop-gradient operator, and β is a hyperparameter for the third commitmentloss .
Revised: Here, sg denotes the stop-gradient operator, and β is a hyperparameter for the third commitment loss .

Original: The first term is a reconstruction loss to estimate the error between the observed x and thereconstructed ˆ x .
Revised: The first term is a reconstruction loss to estimate the error between the observed x and the reconstructed ˆ x .

Original: The second term is the codebook loss to optimize the entries in the codebook.
Revised: The second term is the codebook loss to optimize the entries in the codebook.

Original: Weuse the released VQGAN implementation as our baseline.
Revised: We use the released VQGAN implementation as our baseline.

Original: [Figure]
Revised: [Figure]

Original: Spatially conditional normalization.
Revised: Spatially conditional normalization.

Original: The quantization operator is lossy [31], and similar patchesare often embedded as the same codebook indices, resulting in a repeated artifact when they aresynthesized through existing decoder architectures (see Fig.
Revised: The quantization operator is lossy [31], and similar patches are often embedded as the same codebook indices, resulting in a repeated artifact when they are synthesized through existing decoder architectures (see Fig.

Original: 3 (a)).
Revised: 3 (a)).

Original: As opposed to existing methodsthat directly feed discrete feature maps into the decoder as activations, in this work, we introduce anew spatially conditional normalization layer to propagate the embedded contents to the activations.
Revised: As opposed to existing methods that directly feed discrete feature maps into the decoder as activations, in this work, we introduce a new spatially conditional normalization layer to propagate the embedded contents to the activations.

Original: The key motivation behind this is to add spatial variants to the discrete maps, such that the model cangenerate plausible and diverse results, even for the same quantization index in neighboring regions.
Revised: The key motivation behind this is to add spatial variants to the discrete maps, such that the model can generate plausible and diverse results, even for the same quantization index in neighboring regions.

Original: The structure of our modulated decoder is illustrated in Fig.
Revised: The structure of our modulated decoder is illustrated in Fig.

Original: 2 (left), with an activation F normalizedby a conventional normalization, and then modulated by the learned scale and bias calculated fromthe embedded vector.
Revised: 2 (left), with an activation F normalized by a conventional normalization, and then modulated by the learned scale and bias calculated from the embedded vector.

Original: Specifically, the activation F of the i -th layer in the decoder G θ is given by:
Revised: Specifically, the activation F of the i -th layer in the decoder G θ is given by:

Original: [Equation]
Revised: [Equation]

Original: where F i − 1 is the intermediate feature map, which can be initialized as the positional embedding[12, 40], learned constant [20], or Fourier features [36, 19] for F 0 in the decoder.
Revised: where F i − 1 is the intermediate feature map, which can be initialized as the positional embedding [12, 40], learned constant [20], or Fourier features [36, 19] for F 0 in the decoder.

Original: µ ( · ) and σ ( · )respectively denote the functions for calculating the mean and standard deviation of the activation.
Revised: µ ( · ) and σ ( · ) respectively denote the functions for calculating the mean and standard deviation of the activation.

Original: There are many choices of the normalization.
Revised: There are many choices of the normalization.

Original: Following the baseline VQGAN [11], here we directlyuse the Group Normalization [43].
Revised: Following the baseline VQGAN [11], here we directly use the Group Normalization [43].

Original: ϕ γ ( · ) and ϕ β ( · ) are two learned affine transformations, which areimplemented as 1 × 1 convolutional filters in our setting, to convert the discrete representation z q tothe scaling and bias values.
Revised: ϕ γ ( · ) and ϕ β ( · ) are two learned affine transformations, which are implemented as 1 × 1 convolutional filters in our setting, to convert the discrete representation z q to the scaling and bias values.

Original: Note that, the output of ϕ γ ( · ) and ϕ β ( · ) hold the same resolution to thecurrent activation F i − 1 , which injects spatial variances into the discrete feature.
Revised: Note that, the output of ϕ γ ( · ) and ϕ β ( · ) hold the same resolution to the current activation F i − 1 , which injects spatial variances into the discrete feature.

Original: In fact, this spatially conditional normalization is derived from the adaptive instance normalization(AdaIN), which has been applied for many image synthesis approaches.
Revised: In fact, this spatially conditional normalization is derived from the adaptive instance normalization (AdaIN), which has been applied for many image synthesis approaches, such as style transfer [9, 17], image-to-image translation [18, 27], and image generation [20, 41].

Original: Compared to these methods,our spatially conditional map is a quantized map, which contains learned compact contents.
Revised: Compared to these methods, our spatially conditional map is a quantized map, which contains learned compact contents.

Original: As suchmodulation operator injects the spatially variant information, it encourages the same quantizationentries to generate plausible and diverse results on different locations.
Revised: As such modulation operator injects the spatially variant information, it encourages the same quantization entries to generate plausible and diverse results on different locations.

Original: Multichannel representation.
Revised: Multichannel representation.

Original: Unlike existing quantizers that map an image into a single-channelindex map, here we convert it into a multichannel index map with the shared codebook to furtherimprove the image quality, similar to [47].
Revised: Unlike existing quantizers that map an image into a single-channel index map, here we convert it into a multichannel index map with the shared codebook to further improve the image quality, similar to [47].

Original: This is inspired by the conventional GAN setting,where 1 × 1 × n c random vector is able to generate photorealistic images with reasonable structure[13, 20, 21], showing that values across the channels contain abundant information.
Revised: This is inspired by the conventional GAN setting, where 1 × 1 × n c random vector is able to generate photorealistic images with reasonable structure [13, 20, 21], showing that values across the channels contain abundant information.

Original: In practice,we first subdivide the encoded continuous feature ˆ z ∈ R h × w × n z along the channel dimensioninto multiple chunks, i.e. ˆ z =  ˆ z (1) , ·
Revised: In practice, we first subdivide the encoded continuous feature ˆ z ∈ R h × w × n z along the channel dimension into multiple chunks, i.e. ˆ z = { ˆ z (1) , ·

Original: · · , ˆ z ( c ) } , ˆ z ( c ) ∈ R h × w × n z /c .
Revised: · · , ˆ z ( c ) } , ˆ z ( c ) ∈ R h × w × n z /c .

Original: Each of these chunks is thenquantized based on the equation 2 to the closest codevectors in the codebook.
Revised: Each of these chunks is then quantized based on the equation 2 to the closest codevectors in the codebook.

Original: For 256 × 256 images, we downsample it by a fixed factor of 16 to 16 × 16 features.
Revised: For 256 × 256 images, we downsample it by a fixed factor of 16 to 16 × 16 features.

Original: Unless otherwisenoted, in this work we employ c = 4 parallel chunks, and our equivalent sequence representationis 16 × 16 × 4 , which is 192x times smaller than the original image.
Revised: Unless otherwise noted, in this work we employ c = 4 parallel chunks, and our equivalent sequence representation is 16 × 16 × 4 , which is 192x times smaller than the original image.

Original: Note that, in such way thecodevector dimensionality in the codebook will be reduced to n q = n z /c .
Revised: Note that, in such way the codevector dimensionality in the codebook will be reduced to n q = n z /c .

Original: Due to this reduceddimension of each codevector, the total computational cost is similar to that of the single-channelrepresentation with fully dimensionality.
Revised: Due to this reduced dimension of each codevector, the total computational cost is similar to that of the single-channel representation with fully dimensionality.

Original: In fact, our codebook Z ∈ R K × n q has a smaller size withthe number of dimensions to be n q = n z /c
Revised: In fact, our codebook Z ∈ R K × n q has a smaller size with the number of dimensions to be n q = n z /c

Original: = 256 / 4 = 64 .
Revised: = 256 / 4 = 64 .

Original: While a higher resolution representation, e.g. 32 × 32 × 1 or 64 × 64 × 1 , can also improve thereconstruction quality as in [30, 32, 45], the computational cost will be expensive for the secondstage due to the longer sequence s = h × w (see Sec. 3.2 for details).
Revised: While a higher resolution representation, e.g. 32 × 32 × 1 or 64 × 64 × 1 , can also improve the reconstruction quality as in [30, 32, 45], the computational cost will be expensive for the second stage due to the longer sequence s = h × w (see Sec. 3.2 for details).

Original: In contrast, our multichannelrepresentation maintains a smaller sequence length, i.e. 16 × 16 , although each sequence tokennow is recomposed by four pieces, each of which corresponds to one quantization index.
Revised: In contrast, our multichannel representation maintains a smaller sequence length, i.e. 16 × 16 , although each sequence token now is recomposed by four pieces, each of which corresponds to one quantization index.

Original: Withsuch an exponential combination capacity of K c for each spatial grid feature ˆ z ij , the multichannelrepresentation power becomes much larger than the K entries in the original codebook.
Revised: With such an exponential combination capacity of K c for each spatial grid feature ˆ z ij , the multichannel representation power becomes much larger than the K entries in the original codebook.

Original: 3.2 Modeling Prior Distribution
Revised: 3.2 Modeling Prior Distribution

Original: While a decoder can invert the discrete image embeddings z q to produce images x , we need to tame amodel to estimate the underlying prior distribution over the discrete space to enable image generation.
Revised: While a decoder can invert the discrete image embeddings z q to produce images x , we need to tame a model to estimate the underlying prior distribution over the discrete space to enable image generation.

Original: Since our main goal is to improve the codebook learning stage, we directly employ the existingsetup for the second stage, including the conventional autoregressive model and the latest maskedgenerative image transformer (MaskGIT) [2].
Revised: Since our main goal is to improve the codebook learning stage, we directly employ the existing setup for the second stage, including the conventional autoregressive model and the latest masked generative image transformer (MaskGIT) [2].

Original: Autoregressive token generation.
Revised: Autoregressive token generation.

Original: After embedding an image x into an index sequence s ={ s 1 , · · · , s h × w } of the codebook entries z q , the image generation can be naturally formulated as anautoregressive next-symbol predication problem.
Revised: After embedding an image x into an index sequence s = { s 1 , · · · , s h × w } of the codebook entries z q , the image generation can be naturally formulated as an autoregressive next-symbol predication problem.

Original: In particular, we maximize the likelihood function:
Revised: In particular, we maximize the likelihood function:

Original: [Equation]
Revised: [Equation]

Original: where p ( s i | s <i ) is the probability of having the next symbol s
Revised: where p ( s i | s <i ) is the probability of having the next symbol s

Original: i given all the previous symbols s <i .
Revised: i given all the previous symbols s <i .

Original: Then, the training objective for the second stage is equal to minimize the negative log-likelihood ofthe whole sequence, i.e. L = E x ∼ p ( x ) [ − log p ( s )] .
Revised: Then, the training objective for the second stage is equal to minimize the negative log-likelihood of the whole sequence, i.e. L = E x ∼ p ( x ) [ − log p ( s )] .

Original: There are many choices of autoregressive generation models, such as CNN-based PixelCNN [38]and transformer-based GPT [3].
Revised: There are many choices of autoregressive generation models, such as CNN-based PixelCNN [38] and transformer-based GPT [3].

Original: While they can sequentially produce diverse and reasonable resultsbased on the previously generated results, the sampling process is very slow.
Revised: While they can sequentially produce diverse and reasonable results based on the previously generated results, the sampling process is very slow.

Original: Masked token generation.
Revised: Masked token generation.

Original: To produce the index sequence efficiently, we follow the latest MaskGITto simultaneously predict all tokens in parallel.
Revised: To produce the index sequence efficiently, we follow the latest MaskGIT to simultaneously predict all tokens in parallel.

Original: Inspired by BERT [5], the model aims to estimate thedistribution of all indices based on the visible indices in a fixed length:
Revised: Inspired by BERT [5], the model aims to estimate the distribution of all indices based on the visible indices in a fixed length:

Original: [Equation]
Revised: [Equation]

Original: where s ¯ m denotes conditional representation with partially visible codebook indices.
Revised: where s ¯ m denotes conditional representation with partially visible codebook indices.

Original: In practice,the masked map s ¯ m , with random mask ratios during training, is fed into a multi-layer bidirectionaltransformer to estimate the possible distribution of each masked indices, where the negative loglikelilood objective as above is also calculated between the ground-truth one-hot index and predictedindex.
Revised: In practice, the masked map s ¯ m , with random mask ratios during training, is fed into a multi-layer bidirectional transformer to estimate the possible distribution of each masked indices, where the negative loglikelilood objective as above is also calculated between the ground-truth one-hot index and predicted index.

Original: At inference time, the model runs fixed steps and predicts all tokens simultaneously in parallelbased on the previous produced visible tokens at each iteration.
Revised: At inference time, the model runs fixed steps and predicts all tokens simultaneously in parallel based on the previous produced visible tokens at each iteration.

Original: Then, the most confident tokens areselected out and the remaining masked tokens will be predicted in the next step condition on previouspredicted tokens.
Revised: Then, the most confident tokens are selected out and the remaining masked tokens will be predicted in the next step condition on previous predicted tokens.

Original: We refer readers to MaskGIT [2] for more details.
Revised: We refer readers to MaskGIT [2] for more details.

Original: Multichannel token generation.
Revised: Multichannel token generation.

Original: Since our MoVQ embeds an image into a multichannel sequences = { s 1 , · · · , s h × w } , where s i ∈ { 0 , · · · , |Z| − 1 } c , for each position, we inversely concatenatechunks along the channel as one token for the input, and predict c indices for the output.
Revised: Since our MoVQ embeds an image into a multichannel sequence s = { s 1 , · · · , s h × w } , where s i ∈ { 0 , · · · , |Z| − 1 } c , for each position, we inversely concatenate chunks along the channel as one token for the input, and predict c indices for the output.

Original: In practice,we independently mask each index for various channels and positions.
Revised: In practice, we independently mask each index for various channels and positions.

Original: Compared to MaskGIT, thepredicated indices in our model condition on three directional dependencies, i.e. visible indices fromdifferent positions and channels, resulting in larger diversity due to the combination capability. 
Revised: Compared to MaskGIT, the predicated indices in our model condition on three directional dependencies, i.e. visible indices from different positions and channels, resulting in larger diversity due to the combination capability. [Table]

Original:  Experiments 
Revised:  Experiments 4.1 Experimental Details

Original: 4.1 Experimental Details

Original: Datasets.
Revised: Datasets.

Original: To evaluate the proposed method, we instantiated MoVQ on both unconditional andclass-conditional image generation tasks, with FFHQ [20] and ImageNet [33] respectively.
Revised: To evaluate the proposed method, we instantiated MoVQ on both unconditional and class-conditional image generation tasks, with FFHQ [20] and ImageNet [33] respectively.

Original: Thetraining and validation setting followed the default one of our baseline model VQGAN [11].
Revised: The training and validation setting followed the default one of our baseline model VQGAN [11].

Original: Wetrained all [Table]
Revised: We trained all the models on 256 × 256 images.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: For image generation, we used the two most common evaluation metrics
Revised: For image generation, we used the two most common evaluation metrics

Original: Fréchet Inception Distance ( FID ) [15] and Inception Score ( IS ) [34] to evaluate the quality anddiversity between generated images and ground truth images.
Revised: Fréchet Inception Distance ( FID ) [15] and Inception Score ( IS ) [34] to evaluate the quality and diversity between generated images and ground truth images.

Original: Since we focused on the first stage,we also evaluated the quality between reconstructed images and original images.
Revised: Since we focused on the first stage, we also evaluated the quality between reconstructed images and original images.

Original: Except for therFID score, we additionally reported the traditional patch-level image quality metrics, including peaksignal-to-noise ratio (PSNR), structure similarity index (SSIM), and the latest learned feature-level
Revised: Except for the rFID score, we additionally reported the traditional patch-level image quality metrics, including peak signal-to-noise ratio (PSNR), structure similarity index (SSIM), and the latest learned feature-level

Original: LPIPS [46] for the paired reconstructed images and ground truth images.
Revised: LPIPS [46] for the paired reconstructed images and ground truth images.

Original: 4.2 Image Quantization
Revised: 4.2 Image Quantization

Original: Network structures and implementation details.
Revised: Network structures and implementation details.

Original: For each dataset, we only trained a singlescale quantizer with a codebook Z ∈ R 1024 × 64 , i.e. 1024 codevectors each with 64 dimensions,on 256 × 256 images for all experiments.
Revised: For each dataset, we only trained a single scale quantizer with a codebook Z ∈ R 1024 × 64 , i.e. 1024 codevectors each with 64 dimensions, on 256 × 256 images for all experiments.

Original: Our encoder-decode pipeline is built upon the original
Revised: Our encoder-decoder pipeline is built upon the original

Original: VQGAN 1 , except the only difference that we replaced the original Group Normalization with theproposed spatially conditional normalization layer.
Revised: VQGAN 1 , except the only difference that we replaced the original Group Normalization with the [Table] proposed spatially conditional normalization layer.

Original: We applied the new normalization layer in thefirst three blocks of the decoder.
Revised: We applied the new normalization layer in the first three blocks of the decoder.

Original: Following the default setting in VQGAN, images are alwaysdownsampled by a fixed factor of 16, i.e. from 256 × 256 × 3 to a grid of tokens with the size of16 × 16 × 4 .
Revised: Following the default setting in VQGAN, images are always downsampled by a fixed factor of 16, i [Table] e. from 256 × 256 × 3 to a grid of tokens with the size of 16 × 16 × 4 .

Original: We set hyper-parameters following the baseline VQGAN work, and we trained allmodels with a batch size of 48 across 4 Tesla V100 GPUs with 40 epochs for this stage.
Revised: We set hyper-parameters following the baseline VQGAN work [Table] and we trained all models with a batch size of 48 across 4 Tesla V100 GPUs with 40 epochs for this stage.

Original: We compare MoVQ with the state-of-the-art methods for image reconstruction in Table 1.
Revised: We compare MoVQ with the state-of-the-art methods for image reconstruction in Table 1.

Original: Allinstantiations of our model outperform the state-of-the-art methods under the same compressionratio (192x).
Revised: All instantiations of our model outperform the state-of-the-art methods under the same compression ratio (192x).

Original: This includes  ViT-VQGAN [45] and RQ-VAE [24], which utilize higher resolution[Table] representation and recursively residual representation, respectively.
Revised: This includes the concurrent works ViT-VQGAN [45] and RQ-VAE [24], which utilize higher resolution representation and recursively residual representation, respectively.

Original: Without bells and whistles,though we use a much smaller number of parameters (82.7M), similar to VQGAN (72.1M), MoVQoutperformers ViT-VQGAN [45], which employs a larger transformer model (599M) on higherresolution representation ( 32 × 32 ) for the first stage.
Revised: Without bells and whistles, though we use a much smaller number of parameters (82.7M), similar to VQGAN (72.1M), MoVQ outperformers ViT-VQGAN [45], which employs a larger transformer model (599M) on higher resolution representation ( 32 × 32 ) for the first stage.

Original: The concurrent RQ-VAE work [24] alsorepresents an image with multiple channels by recursively calculating the residual informationbetween quantized vectors and their continuous ones, which requires much more embedding times.
Revised: The concurrent RQ-VAE work [24] also represents an image with multiple channels by recursively calculating the residual information between quantized vectors and their continuous ones, which requires much more embedding times.

Original: Furthermore, the entries in RQ-VAE are not equally important because the residual representationhighlights the codevectors in the first round, leaving the other codevectors to capture the small residualinformation.
Revised: Furthermore, the entries in RQ-VAE are not equally important because the residual representation highlights the codevectors in the first round, leaving the other codevectors to capture the small residual information.

Original: In contrast, our codevectors share the same significance in all channels, resulting inlarger representation capability.
Revised: In contrast, our codevectors share the same significance in all channels, resulting in larger representation capability.

Original: More importantly, our model dramatically improves the reconstructedimages quality on all metrics, suggesting the reconstructed images are closer to the original inputs,which contributes to more downstream image interpolation tasks.
Revised: More importantly, our model dramatically improves the reconstructed images quality on all metrics, suggesting the reconstructed images are closer to the original inputs, which contributes to more downstream image interpolation tasks.

Original: Note that, our learned codebook contains only 1024 codevectors with 64 dimensionality, but interestingly outperforms other methods using larger codebook sizes.
Revised: Note that, our learned codebook contains only 1024 codevectors with 64 dimensionality, but interestingly outperforms other methods using larger codebook sizes.

Original: This suggests that a better quantizercan improve the codebook usage, and it is not necessary to greedily increase the codebook size.
Revised: This suggests that a better quantizer can improve the codebook usage, and it is not necessary to greedily increase the codebook size.

Original: The qualitative results are visualized in Figs.
Revised: The qualitative results are visualized in Figs.

Original: 3 and 4.
Revised: 3 and 4.

Original: MoVQ achieves impressive results undervarious conditions.
Revised: MoVQ achieves impressive results under various conditions.

Original: In Fig.
Revised: In Fig.

Original: 3, we compare our MoVQ with the baseline model VQGAN [11] and theconcurrent model RQ-VAE 
Revised: 3, we compare our MoVQ with the baseline model VQGAN [11] and the concurrent model RQ-VAE [24].

Original: VQGAN holds repeated artifacts on the similar semantic patches,such as the grass and trees.
Revised: VQGAN holds repeated artifacts on the similar semantic patches, such as the grass and trees.

Original: RQ-VAE improves the visual appearance, but exhibits systematic artifactswith lossy information.
Revised: RQ-VAE improves the visual appearance, but exhibits systematic artifacts with lossy information.

Original: Our MoVQ shows no such artifacts, providing much more realistic details.
Revised: Our MoVQ shows no such artifacts, providing much more realistic details.

Original: 4.3 Image Generation
Revised: 4.3 Image Generation

Original: Network structures and implementation details.
Revised: Network structures and implementation details.

Original: After embedding images as a sequence, atransformer is implemented to estimate the underlying prior distribution in the second stage.
Revised: After embedding images as a sequence, a transformer is implemented to estimate the underlying prior distribution in the second stage.

Original: Here, weused the same configuration for all models: 24 layers, 16 attention heads, 1024 embedding dimensionsand 4096 hidden dimensions.
Revised: Here, we used the same configuration for all models: 24 layers, 16 attention heads, 1024 embedding dimensions and 4096 hidden dimensions.

Original: The network architecture is build upon the VQGAN baseline, exceptthat we predicted a 16 × 16 × 4 × 1024 tensor in the final layer, where 16 × 16 × 4 is the number ofpredicted indexes, and 1024 is the number of entries in our learned codebook.
Revised: The network architecture is build upon the VQGAN baseline, except that we predicted a 16 × 16 × 4 × 1024 tensor in the final layer, where 16 × 16 × 4 is the number of predicted indexes, and 1024 is the number of entries in our learned codebook.

Original: Here, we set the imagegeneration in two scenarios: (1) an " auto " scenario, in which the training and inference are totallysimilar with the VQGAN baseline, and (2) a " mask " scenario, in which the training and inferenceis inspired by MaskGIT [2] 2 .
Revised: Here, we set the image generation in two scenarios: (1) an " auto " scenario, in which the training and inference are totally similar with the VQGAN baseline, and (2) a " mask " scenario, in which the training and inference [Table] is inspired by MaskGIT [2] 2 .

Original: Unless otherwise noted, we default samples tokens as in MaskGITfor much faster sampling.
Revised: Unless otherwise noted [Table] we default samples tokens as in MaskGIT for much faster sampling.

Original: The hyper-parameters follow the defaults setting as in VQGAN, and wetrained all models with a batch size of 64 across 4 Tesla V100 GPUs with 200 epochs.
Revised: The hyper-parameters follow the defaults setting as in VQGAN [Table] and we trained all models with a batch size of 64 across 4 Tesla V100 GPUs with 200 epochs [Table]

Original: The unconditional generation results are compared in Table 2.
Revised: The unconditional generation results are compared in Table 2.

Revised: While the original autoregressive sampling in our baseline VQGAN [11] provides tiny better generation results

Revised: [Table]

Revised: the sampling time is much more expensive.

Original: Following MaskGIT, we do not set anyspecial sampling strategies such as top- k and top- p sampling heuristics.
Revised: Following MaskGIT, we do not set any special sampling strategies such as top- k and top- p sampling heuristics.

Original: We produce 60,000 samplesat the inference time.
Revised: We produce 60,000 samples at the inference time.

Original: Our model outperforms most VQ-based methods, with smaller parameters andfaster inference time.
Revised: Our model outperforms most VQ-based methods, with smaller parameters and faster inference time.

Original: While our model performs worse than the concurrent work ViT-VQGAN, theyuse a much larger model (5 times than ours) in the second stage with a longer training, which is not a[Table] [Figure] directly fair comparison.
Revised: While our model performs worse than the concurrent work ViT-VQGAN, they use a much larger model ( times than ours) in the second stage with a longer training, which is not a directly fair comparison.

Original: A few generated samples are visualized in Figures 1 (see the Appendix formore).
Revised: A few generated samples are visualized in Figures 1 (see the Appendix for more).

Original: MoVQ generates photorealistic high-fidelity images, with a large diversity.
Revised: MoVQ generates photorealistic high-fidelity images, with a large diversity.

Original: We also compare the proposed MoVQ with state-of-the-art models for class-conditional imagegeneration on ImageNet 256 × 256 .
Revised: We also compare the proposed MoVQ with state-of-the-art models for class-conditional image generation on ImageNet 256 × 256 .

Original: Our model generates 50 samples for each category, 50,000samples in total, for the quantitative evaluation.
Revised: Our model generates 50 samples for each category, 50, samples in total, for the quantitative evaluation.

Original: As shown in Table 3, the proposed MoVQ significantlyimprove the performance over the baseline model VQGAN [11].
Revised: As shown in Table 3, the proposed MoVQ significantly improve the performance over the baseline model VQGAN [11].

Original: Compared with the concurrent work
Revised: Compared with the concurrent work

Original: RQ-VAE [24], which also employs a multi-channel representation, our performance is competitivewith them, even without the specially designed RQ-transformer for further predicting multichannelindexes.
Revised: RQ-VAE [24], which also employs a multi-channel representation , our performance is competitive with them, even without the specially designed RQ-transformer for further predicting multichannel indexes.

Original: As MaskGIT and VIT-VQGAN use more GPUs for longer training, it is not a directlyfair comparison with our MoVQ.
Revised: As MaskGIT and VIT-VQGAN use more GPUs for longer training, it is not a directly fair comparison with our MoVQ.

Original: Figures 5 show a few generated samples.
Revised: Figures 5 show a few generated samples.

Original: Note that, the proposed
Revised: Note that, the proposed

Original: MoVQ appears to be able to generate high frequency details, such as the eyes of the bird.
Revised: MoVQ appears to be able to generate high frequency details, such as the eyes of the bird.

Original: 4.4 Discussion
Revised: 4.4 Discussion

Original: We run a few ablations to analyze the proposed MoVQ.
Revised: We run a few ablations to analyze the proposed MoVQ.

Original: The results are summarized in Fig.
Revised: The results are summarized in Fig.

Original: Architecture.
Revised: Architecture.

Original: Fig.
Revised: Fig.

Original: 6 (a) reports MoVQ with various decoder settings.
Revised: 6 (a) reports MoVQ with various decoder settings.

Original: Compared with the baselinemodel VQGAN ( A ), although we used the same settings for the network architecture, the multichannelrepresentation ( B ) naturally leads to a significant improvement by decomposing and recomposing thefeatures in channel.
Revised: Compared with the baseline model VQGAN ( A ), although we used the same settings for the network architecture, the multichannel representation ( B ) naturally leads to a significant improvement by decomposing and recomposing the features in channel.

Original: For the spatially conditional normalization ( C ), we explored three most commonlyused embedding functions as the initial feature F 0 , including positional embedding [12, 40], learnedconstant [20], and Fourier features [36, 19].
Revised: For the spatially conditional normalization ( C ), we explored three most commonly used embedding functions as the initial feature F 0 , including positional embedding [12, 40], learned constant [20], and Fourier features [36, 19].

Original: For positional embedding, we implemented sine andcosine functions of different frequencies on different locations and channels.
Revised: For positional embedding, we implemented sine and cosine functions of different frequencies on different locations and channels.

Original: The learned constant isimplemented as learned parameters with the same size as the embedded discrete feature.
Revised: The learned constant is implemented as learned parameters with the same size as the embedded discrete feature.

Original: However,the improvement from both methods is limited.
Revised: However, the improvement from both methods is limited.

Original: We believe a key reason is the lack of special identity 
Revised: We believe a key reason is the lack of special identity information for the reconstruction task, while they can provide high frequency signals. In contrast, each Fourier feature is calculated from the corresponding discrete feature, which contains the identity feature, as well as the high-frequency signals, resulting in a significant improvement. Goldfish Goldfinch

Revised: Goose

Revised: Flamingo

Original: Methods rFID FID
Revised: Silky Terrier Tibetan Mastiff Red Panda Grey Fox

Revised: Codebook size.

Revised: Vanilla VQGAN shows a larger codebook leads to a much better representation.

Revised: However, this is not true in our experimental results.

Revised: In Fig.

Revised: 6 (b) we compare the results with different codebook sizes.

Revised: As shown in the figure, the difference between the 1024 codebook entries and 16384 entries in our setting is negligible.

Revised: This suggests that once a better quantizer is designed, it is sufficient to represent vast amounts of images using a small codebook, which makes the model easier to train.

Revised: Besides, we found a low codebook usage in current VQ models, while our design can somehow improve the codebook usage.

Revised: We plan to investigate this issue in future work.

Revised: Methods rFID FID

Original: A Baseline VQGAN 4.42 11.4 B + multichannel x 3.78 10.
Revised: A Baseline VQGAN 4.42 11.4 B + multichannel x 3.78 10.

Original: C w/ sinusiods 3.52 9.
Revised: C w/ sinusiods 3.52 9.

Original: w/ learned constants 3.48 8.
Revised: w/ learned constants 3.48 8.

Original: w/ Fourier features 2.26 8.
Revised: w/ Fourier features 2.26 8.

Original: Methods Num Z rFID
Revised: Methods Num Z rFID

Original: VQGAN6.
Revised: VQGAN6.

Original: VQGAN3.
Revised: VQGAN3.

Original: MoVQ 1. MoVQ 1.
Revised: MoVQ 1. MoVQ 1.

Original: 16 channels 1.
Revised: 16 channels 1.

Original: r F I D
Revised: r F I D

Original: Reconstruction FID Generation FID 8.8 8.
Revised: Reconstruction FID Generation FID 8.8 8.

Original: F I D [Figure] spatially 
Revised: F I D a spatially conditional normalization

Original: [Figure]
Revised: [Figure]

Original: information for the reconstruction task, while they can provide high frequency signals.

Original: In contrast,each Fourier feature is calculated from the corresponding discrete feature, which contains the identityfeature, as well as the high-frequency signals, resulting in a significant improvement.

Original: Codebook size.

Original: Vanilla VQGAN shows a larger codebook leads to a much better representation.

Original: However, this is not true in our experimental results.

Original: In Fig.

Original: 6 (b) we compare the results withdifferent codebook sizes.

Original: As shown in the figure, the difference between the 1024 codebook entriesand 16384 entries in our setting is negligible.

Original: This suggests that once a better quantizer is designed, itis sufficient to represent vast amounts of images using a small codebook, which makes the modeleasier to train and evaluation.

Original: Besides, we found a low codebook usage in current VQ models, whileour design can somehow improve the codebook usage.

Original: Latent size.
Revised: Latent size.

Original: Vector quantization is a compression system, and we exploit the influence of differentcompress ratios by using different latent sizes.
Revised: Vector quantization is a compression system, and we exploit the influence of different compress ratios by using different latent sizes.

Original: In order not to dramatically increase the computationalcost, we mainly compare the performance of the latent representation with different numbers ofchannels in Fig.
Revised: In order not to dramatically increase the computational cost, we mainly compare the performance of the latent representation with different numbers of channels in Fig.

Original: 6 (c).
Revised: 6 (c).

Original: The reconstruction performance is naturally benefits from more channels dueto the lower compression ratio.
Revised: The reconstruction performance is naturally benefits from more channels due to the lower compression ratio.

Original: However, we note that not all configurations automatically benefitfrom more channels on image generation, as we need to predict more information in the secondstage.
Revised: However, we note that not all configurations automatically benefit from more channels on image generation, as we need to predict more information in the second stage.

Original: Thus, we select a trade-off between the reconstruction and the generation.
Revised: Thus, we select a trade-off between the reconstruction and the generation.

Original: It is worth to notethat this is an initial step toward combining spatial vectors in the likelihood model with the channelinformation as in GAN for the generation task.
Revised: It is worth to note that this is an initial step toward combining spatial vectors in the likelihood model with the channel information as in GAN for the generation task.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we introduced MoVQ, a new model that is simple, efficient yet effective for generatingdiverse and plausible images using a better quantizer with a powerful transformer to estimating theprior.
Revised: In this work, we introduced MoVQ, a new model that is simple, efficient yet effective for generating diverse and plausible images using a better quantizer with a powerful transformer to estimating the prior.

Original: Our encoder and decoder architectures are kept simple as in the baseline framework VQGAN,except that we incorporate a spatially conditional normalization and multichannel latent maps into the
Revised: Our encoder and decoder architectures are kept simple as in the baseline framework VQGAN, except that we incorporate a spatially conditional normalization and multichannel latent maps into the

Original: VQGAN method.
Revised: VQGAN method.

Original: Experimental results show that MoVQ significantly outperforms the state-of-the-art
Revised: Experimental results show that MoVQ significantly outperforms the state-of-the-art

Original: VQ models on image modeling, yet without increasing computational cost.
Revised: VQ models on image modeling, yet without increasing computational cost.

Original: With a better quantizer,we show that the fidelity of our unconditional samples and class-conditional samples are better thanthe existing methods.
Revised: With a better quantizer, we show that the fidelity of our unconditional samples and class-conditional samples are better than the existing methods.

Original: In the future, we would like to explore the semantic meaning of each entry in the learned codebook.
Revised: In the future, we would like to explore the semantic meaning of each entry in the learned codebook.

Original: As our model not only generates high fidelity image as the state-of-the-art GAN, but also providesmuch better reconstructed images, we are excited about the future of VQ-based models and plan toapply them to more image inversion, interpolation and translation tasks.
Revised: As our model not only generates high fidelity image as the state-of-the-art GAN, but also provides much better reconstructed images, we are excited about the future of VQ-based models and plan to apply them to more image inversion, interpolation and translation tasks.

Original: Limitation.
Revised: Limitation.

Original: Although our MoVQ dramatically improves the image representation quality than thestate-of-the-art under the same compression ratio, the model sometimes generates images with ahigh-frequency appearance, while the structure information is missing.
Revised: Although our MoVQ dramatically improves the image representation quality than the state-of-the-art under the same compression ratio, the model sometimes generates images with a high-frequency appearance, while the structure information is missing.

Original: We believe this is partiallydue to the multichannel representation we employed in our model.
Revised: We believe this is partially due to the multichannel representation we employed in our model.

Original: Therefore, a better generationmodel for modeling multichannel indexes needs to be further studied.
Revised: Therefore, a better generation model for modeling multichannel indexes needs to be further studied.

Revised: Acknowledgements: This research was supported by Monash FIT research Grant.

Original: Checklist
Revised: Checklist

Original:  For all authors...
Revised: The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] or [N/A] . You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: For example: • Did you include the license to the code and datasets? [Yes] See Section.• Did you include the license to the code and datasets?

Original: contributions and scope?
Revised: [No] The code and the data are proprietary. • Did you include the license to the code and datasets? [N/A] Please do not modify the questions and only use the provided macros for your answers.

Original: [Yes]
Revised: Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below. For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original:  (b) Did you describe the limitations of your work?[Yes]
Revised: [Yes] (b) Did you describe the limitations of your work?

Original:  (c) Did you discuss any potential negative societal impacts of your work?[N/A]
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?[N/A]
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original:  (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on [Table]
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]