Original: MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
Revised: MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Although two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images, their quantization operator encodessimilar patches within an image into the same index, resulting in a repeated artifact for similar adjacent regions using existing decoder architectures.
Revised: Although two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images, their quantization operator encodessimilar patches within an image into the same index, resulting in a repeated artifact for similar adjacent regions using existing decoder architectures.

Original: To addressthis issue, we propose to incorporate the spatially conditional normalization tomodulate the quantized vectors so as to insert spatially variant information to theembedded index maps, encouraging the decoder to generate more photorealisticimages.
Revised: To addressthis issue, we propose to incorporate the spatially conditional normalization tomodulate the quantized vectors so as to insert spatially variant information to theembedded index maps, encouraging the decoder to generate more photorealisticimages.

Original: Moreover, we use multichannel quantization to increase the recombinationcapability of the discrete codes without increasing the cost of model and codebook.
Revised: Moreover, we use multichannel quantization to increase the recombinationcapability of the discrete codes without increasing the cost of model and codebook.

Original: Additionally, to generate discrete tokens at the second stage, we adopt a Masked
Revised: Additionally, to generate discrete tokens at the second stage, we adopt a Masked

Original: Generative Image Transformer (MaskGIT) to learn an underlying prior distributionin the compressed latent space, which is much faster than the conventional autoregressive model.
Revised: Generative Image Transformer (MaskGIT) to learn an underlying prior distributionin the compressed latent space, which is much faster than the conventional autoregressive model.

Original: Experiments on two benchmark datasets demonstrate that ourproposed modulated VQGAN is able to greatly improve the reconstructed imagequality as well as provide high-fidelity image generation.
Revised: Experiments on two benchmark datasets demonstrate that ourproposed modulated VQGAN is able to greatly improve the reconstructed imagequality as well as provide high-fidelity image generation.

Original:  Introduction
Revised:  Introduction

Original: The vision community has rapidly improved image synthesis results on quality, diversity and resolution in the past few years.
Revised: The vision community has rapidly improved image synthesis results on quality, diversity and resolution in the past few years.

Original: Many powerful baseline frameworks have been introduced, such as
Revised: Many powerful baseline frameworks have been introduced, such as

Original: Generative Adversarial Networks (GAN) [13], Variational AutoEncoder (VAE) [23], Flow-based
Revised: Generative Adversarial Networks (GAN) [13], Variational AutoEncoder (VAE) [23], Flow-based

Original: Generators [8] and Diffusion Probabilistic Models (DPM) [35].
Revised: Generators [8] and Diffusion Probabilistic Models (DPM) [35].

Original: These methods are conceptually intuitive, leading to an explosion of image synthesis works that push the boundaries of image generationquality and diversity [14, 1, 22, 20, 21, 37, 4].
Revised: These methods are conceptually intuitive, leading to an explosion of image synthesis works that push the boundaries of image generationquality and diversity [14, 1, 22, 20, 21, 37, 4].

Original: Among the existing image synthesis frameworks, Vector Quantized-Variational AutoEncoder (VQVAE) [39] is a very popular nova, which learns a compressed discrete representation for each imageat the first stage and subsequently learns the underlying prior distribution in the discrete latent spaceat the second stage.
Revised: Among the existing image synthesis frameworks, Vector Quantized-Variational AutoEncoder (VQVAE) [39] is a very popular nova, which learns a compressed discrete representation for each imageat the first stage and subsequently learns the underlying prior distribution in the discrete latent spaceat the second stage.

Original: Unlike conventional GANs that pursue a fragile balance in a minimax game,resulting in unstable training, the optimization target of VQ-VAE is a definite negative log-likelihood
Revised: Unlike conventional GANs that pursue a fragile balance in a minimax game,resulting in unstable training, the optimization target of VQ-VAE is a definite negative log-likelihood

Original: (NLL) objective of the training data.
Revised: (NLL) objective of the training data.

Original: By maximizing the log-likelihood over all training examples,theoretically VQ-VAE is able to cover all modes of the data, and thus bypass the “mode collapse”issue in GAN.
Revised: By maximizing the log-likelihood over all training examples,theoretically VQ-VAE is able to cover all modes of the data, and thus bypass the “mode collapse”issue in GAN.

Original: On the other hand, compared with VAE [23], VQ-VAE maps an image into a palette oflatent discrete codes in higher resolution with spatial structure information and learns the compositionof the codes from the data itself, which overcomes the long-dragged image quality issue of VAE inimage synthesis.
Revised: On the other hand, compared with VAE [23], VQ-VAE maps an image into a palette oflatent discrete codes in higher resolution with spatial structure information and learns the compositionof the codes from the data itself, which overcomes the long-dragged image quality issue of VAE inimage synthesis.

Original: These advantages of VQ-VAE have led to remarkable image synthesis results asevident by its recent extensions, such as VQ-VAE-2 [31], DALL-E [30], VQGAN [11], ImageBART[10], LDMs [32], VIT-VQGAN [45], RQ-VAE [24], MaskGIT [2] and DALL-E-2 [29].
Revised: These advantages of VQ-VAE have led to remarkable image synthesis results asevident by its recent extensions, such as VQ-VAE-2 [31], DALL-E [30], VQGAN [11], ImageBART[10], LDMs [32], VIT-VQGAN [45], RQ-VAE [24], MaskGIT [2] and DALL-E-2 [29].

Original: Despite the great performance, the VQ-VAE or VQGAN pipeline also has its shortcomings.
Revised: Despite the great performance, the VQ-VAE or VQGAN pipeline also has its shortcomings.

Original: Forexample, its second stage is typically modelled as a sequence generation process in an autoregressiveway [39, 11], i.e. generating each discrete latent code one by one at different spatial locations, whichis very time-consuming for inference.
Revised: Forexample, its second stage is typically modelled as a sequence generation process in an autoregressiveway [39, 11], i.e. generating each discrete latent code one by one at different spatial locations, whichis very time-consuming for inference.

Original: MaskGIT [2] nicely addresses this issue by predicting multipletokens based on the prediction confidence at each step, which greatly reduces the number of stepsneeded in the autoregressive token generator.
Revised: MaskGIT [2] nicely addresses this issue by predicting multipletokens based on the prediction confidence at each step, which greatly reduces the number of stepsneeded in the autoregressive token generator.

Original: In contrast, in this paper we focus on improving the stage-1 learning of the VQ-based imagesynthesis pipeline.
Revised: In contrast, in this paper we focus on improving the stage-1 learning of the VQ-based imagesynthesis pipeline.

Original: Specifically, we notice another inherent shortcoming of the VQ-based methods,i.e. they often generate repeated artifact patterns in image synthesis (see Fig. 3), which is becausethe quantization operator drops away some variances and embeds similar patches into the samequantization index.
Revised: Specifically, we notice another inherent shortcoming of the VQ-based methods,i.e. they often generate repeated artifact patterns in image synthesis (see Fig. 3), which is becausethe quantization operator drops away some variances and embeds similar patches into the samequantization index.

Original: To address this, motivated by the spatially conditional normalization and positionembedding literature [17, 27, 20, 41], we re-design the decoder architecture in a way that utilizesa spatially conditional normalization layer to modulate the activations using the quantized vectors,which essentially adds spatially variant information to the discrete representation.
Revised: To address this, motivated by the spatially conditional normalization and positionembedding literature [17, 27, 20, 41], we re-design the decoder architecture in a way that utilizesa spatially conditional normalization layer to modulate the activations using the quantized vectors,which essentially adds spatially variant information to the discrete representation.

Original: Moreover, weleverage a multichannel representation technique, introduced in [47] for image completion, to keepthe codebook size manageable while with sufficient representation capability for image generation.
Revised: Moreover, weleverage a multichannel representation technique, introduced in [47] for image completion, to keepthe codebook size manageable while with sufficient representation capability for image generation.

Original: Lastly, with a better quantizer, we follow MaskGIT [2] to speed up the prior distribution training overthe compact discrete representations in the second stage [45].
Revised: Lastly, with a better quantizer, we follow MaskGIT [2] to speed up the prior distribution training overthe compact discrete representations in the second stage [45].

Original: In summary, our main contributions are as follows:
Revised: In summary, our main contributions are as follows:

Original: • We point out an inherent shortcoming of the VQ-based image synthesis pipeline: having
Revised: • We point out an inherent shortcoming of the VQ-based image synthesis pipeline: having

Original: repeated artifacts in similar nearby regions, and solve it by introducing a spatially conditional
Revised: repeated artifacts in similar nearby regions, and solve it by introducing a spatially conditional

Original: normalization to provide spatially variant information for different locations.
Revised: normalization to provide spatially variant information for different locations.

Original: • With the help of our proposed spatially conditional normalization, we further incorporate
Revised: • With the help of our proposed spatially conditional normalization, we further incorporate

Original: the multichannel representation that dramatically improves the reconstructed image quality
Revised: the multichannel representation that dramatically improves the reconstructed image quality

Original: using the same encoder-decoder layers as in VQGAN.• When sampled from a learned prior on all directions for multi-locations and multi-channels,experimental results on two benchmark datasets show that our synthesized samples hold notonly high quality but also large diversity.
Revised: using the same encoder-decoder layers as in VQGAN.• When sampled from a learned prior on all directions for multi-locations and multi-channels,experimental results on two benchmark datasets show that our synthesized samples hold notonly high quality but also large diversity.

Original:  Related Work
Revised:  Related Work

Original: VQ-based image synthesis.
Revised: VQ-based image synthesis.

Original: Our model is derived from VQGAN [11], which is built upon the twostage VQ-VAE [39]: first, a quantizer with an encoder-decoder architecture is trained to embed imagesinto compact sequences using discrete tokens from a learned codebook, and then a prior network islearned to model the underlying distribution in the discrete space.
Revised: Our model is derived from VQGAN [11], which is built upon the twostage VQ-VAE [39]: first, a quantizer with an encoder-decoder architecture is trained to embed imagesinto compact sequences using discrete tokens from a learned codebook, and then a prior network islearned to model the underlying distribution in the discrete space.

Original: Unlike the minimax game in popular
Revised: Unlike the minimax game in popular

Original: GAN models [13, 1, 20, 21], the VQ-based generator is trained by optimizing negative log-likelihoodover all examples in the training set, leading to a stable training and bypassing the “mode collapse”issue.
Revised: GAN models [13, 1, 20, 21], the VQ-based generator is trained by optimizing negative log-likelihoodover all examples in the training set, leading to a stable training and bypassing the “mode collapse”issue.

Original: Driven by these advantages, many image synthesis models follow the two-stage paradigm,such as image generation [31, 45, 2, 24, 16], image-to-image translation [11, 10, 32], text-to-imagesynthesis [30, 29, 10, 7], conditional video generation [28, 42, 44], and image completion [11, 10, 47].
Revised: Driven by these advantages, many image synthesis models follow the two-stage paradigm,such as image generation [31, 45, 2, 24, 16], image-to-image translation [11, 10, 32], text-to-imagesynthesis [30, 29, 10, 7], conditional video generation [28, 42, 44], and image completion [11, 10, 47].

Original: Apart from VQGAN, the most related works also include ViT-VQGAN [45] and RQ-VAE [24] thataim to train a better quantizer in the first stage.
Revised: Apart from VQGAN, the most related works also include ViT-VQGAN [45] and RQ-VAE [24] thataim to train a better quantizer in the first stage.

Original: Compared to them, our model is simple and efficient,yet effective to improve the image quality, without adding the computational cost on higher resolutionrepresentations [45] or more stages of recursive quantization [24].
Revised: Compared to them, our model is simple and efficient,yet effective to improve the image quality, without adding the computational cost on higher resolutionrepresentations [45] or more stages of recursive quantization [24].

Original: Spatially conditional normalization, sometimes called adaptive instance normalization, has beenwidely used in image synthesis task [9, 17, 18, 27, 20, 41].
Revised: Spatially conditional normalization, also called adaptive instance normalization, has been widelyused in image synthesis task [9, 17, 18, 27, 20, 41].

Original: Among them, the spatially conditionalinput contains various variants, such as style images [9, 17], target domain images [18], semantic maps [27], learned vectors [20] and random heatmaps [41].
Revised: Among them, the spatially conditional inputcontains various variants, such as style images [9, 17], target domain images [18], semantic maps [27], learned vectors [20] and random heatmaps [41].

Original: Comparing our approach to these works,we obtain the spatially conditional input from the embedded discrete features, which contain learnedcompact contents.
Revised: Comparing our approach to these works, ourspatially conditional input is the embedded discrete features, which contain learned compact contents.

Original: To the best of our knowledge, this is the first work to modulate quantized vectors and use multichannelquantization on the VQ-based image generation framework.
Revised: To the best of our knowledge, this is the first work to modulate quantized vectors and use multichannelquantization on the VQ-based image generation framework.

Original: In the following sections, we willdescribe the modulated quantized vectors and multichannel quantization and discuss their advantagesover the concurrent models such as [32, 45] and [24] in detail.
Revised: In the following sections, we willdescribe the modulated quantized vectors and multichannel quantization and discuss their advantagesover the concurrent models such as [32, 45] and [24] in detail.

Original: [Figure]
Revised: [Figure]

Original: 𝑧𝑧 
Revised: 𝑧𝑧 𝑞𝑞

Original: [Figure]
Revised: [Figure]

Original: 𝐹𝐹 𝛽𝛽 ( ℎ 
Revised: 𝐹𝐹 𝛽𝛽 ( ℎ , 𝑤𝑤 , 𝑐𝑐 ) ( ℎ

Original: [Figure]
Revised: [Figure]

Original: 𝑐𝑐
Revised: 𝑤𝑤

Original: [Figure]
Revised: [Figure]

Revised: 𝑐𝑐 )

Revised: [Figure]

Revised: ( ℎ

Revised: [Figure]

Original: ℎ
Revised: 𝑤𝑤

Original: [Figure]
Revised: [Figure]

Original:  Method
Revised:  Method

Original: The proposed approach, illustrated in Fig.
Revised: The proposed approach, illustrated in Fig.

Original: 2, follows a two-stage recipe that embeds images usinga learnable codebook (see Fig.
Revised: 2, follows a two-stage recipe that embeds images usinga learnable codebook (see Fig.

Original: 2 left), and then tames a bidirectional transformer to estimate theunderlying prior distribution over the discrete latent space (see Fig.
Revised: 2 left), and then tames a bidirectional transformer to estimate theunderlying prior distribution over the discrete latent space (see Fig.

Original: 2 right).
Revised: 2 right).

Original: The key observationwe found is that a better quantizer will naturally lead to better image reconstruction and generationquality [45].
Revised: The key observationwe found is that a better quantizer will naturally lead to better image reconstruction and generationquality [45].

Original: Hence, our main goal in this work is to improve the quantization in the first stage, yetwithout suffering from awkward computational cost as in concurrent works [45, 24].
Revised: Hence, our main goal in this work is to improve the quantization in the first stage, yetwithout suffering from awkward computational cost as in concurrent works [45, 24].

Original: 3.1 Modulating Quantized Vector
Revised: 3.1 Modulating Quantized Vector

Original: Background.
Revised: Background.

Original: Given an image x ∈ R H × W × 3 , vanilla VQ-VAEs learn a discrete codebook to represent observations as a collection of codebook entries z q ∈ R h × w × n q , where n q is the dimensionalityof quantized vectors in the codebook.
Revised: Given an image x ∈ R H × W × 3 , vanilla VQ-VAEs learn a discrete codebook to represent observations as a collection of codebook entries z q ∈ R h × w × n q , where n q is the dimensionalityof quantized vectors in the codebook.

Original: In this way, each image can be equivalently represented asa compact sequence s with h · w indices of the codevectors z q .
Revised: In this way, each image can be equivalently represented asa compact sequence s with h · w indices of the codevectors z q .

Original: Formally, the observed image x isreconstructed by:
Revised: Formally, the observed image x isreconstructed by:

Original: [Equation]
Revised: [Equation]

Original: In particular, an encoder E ψ ( · ) first embeds an image x into a continuous vector ˆ z , and the quantizationoperator q ( · ) is then conducted to transfer the continuous feature ˆ z into the discrete space by lookingup the closest codebook entry z k for each spatial grid feature ˆ z ij within ˆ z :
Revised: In particular, an encoder E ψ ( · ) first embeds an image x into a continuous vector ˆ z , and the quantizationoperator q ( · ) is then conducted to transfer the continuous feature ˆ z into the discrete space by lookingup the closest codebook entry z k for each spatial grid feature ˆ z ij within ˆ z :

Original: [Equation]
Revised: [Equation]

Original: where Z ∈ R K × n q is the codebook that consists of K entries with n q dimensions.
Revised: where Z ∈ R K × n q is the codebook that consists of K entries with n q dimensions.

Original: The quantizedvector z q is finally transmitted to a decoder G θ ( · ) for rebuilding the original image.
Revised: The quantizedvector z q is finally transmitted to a decoder G θ ( · ) for rebuilding the original image.

Original: The overallmodels and the codebook can be learned by optimizing the following objective:
Revised: The overallmodels and the codebook can be learned by optimizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: Here, sg denotes the stop-gradient operator, and β is a hyperparameter for the third commitmentloss .
Revised: Here, sg denotes the stop-gradient operator, and β is a hyperparameter for the third commitmentloss .

Original: The first term is a reconstruction loss to estimate the error between the observed x and thereconstructed ˆ x .
Revised: The first term is a reconstruction loss to estimate the error between the observed x and thereconstructed ˆ x .

Original: The second term is the codebook loss to optimize the entries in the codebook.
Revised: The second term is the codebook loss to optimize the entries in the codebook.

Original: Weuse the released VQGAN implementation as our baseline.
Revised: Weuse the released VQGAN implementation as our baseline.

Original: [Figure]
Revised: [Figure]

Original: Spatially conditional normalization.
Revised: Spatially conditional normalization.

Original: The quantization operator is lossy [31], and similar patchesare often embedded as the same codebook indices, resulting in a repeated artifact when they aresynthesized through existing decoder architectures (see Fig.
Revised: The quantization operator is lossy [31], and similar patchesare often embedded as the same codebook indices, resulting in a repeated artifact when they aresynthesized through existing decoder architectures (see Fig.

Original: 3 (a)).
Revised: 3 (a)).

Original: As opposed to existing methodsthat directly feed discrete feature maps into the decoder as activations, in this work, we introduce anew spatially conditional normalization layer to propagate the embedded contents to the activations.
Revised: As opposed to existing methodsthat directly feed discrete feature maps into the decoder as activations, in this work, we introduce anew spatially conditional normalization layer to propagate the embedded contents to the activations.

Original: The key motivation behind this is to add spatial variants to the discrete maps, such that the model cangenerate plausible and diverse results, even for the same quantization index in neighboring regions.
Revised: The key motivation behind this is to add spatial variants to the discrete maps, such that the model cangenerate plausible and diverse results, even for the same quantization index in neighboring regions.

Original: The structure of our modulated decoder is illustrated in Fig.
Revised: The structure of our modulated decoder is illustrated in Fig.

Original: 2 (left), with an activation F normalizedby a conventional normalization, and then modulated by the learned scale and bias calculated fromthe embedded vector.
Revised: 2 (left), with an activation F normalizedby a conventional normalization, and then modulated by the learned scale and bias calculated fromthe embedded vector.

Original: Specifically, the activation F of the i -th layer in the decoder G θ is given by:
Revised: Specifically, the activation F of the i -th layer in the decoder G θ is given by:

Original: [Equation]
Revised: [Equation]

Original: where F i − 1 is the intermediate feature map, which can be initialized as the positional embedding[12, 40], learned constant [20], or Fourier features [36, 19] for F 0 in the decoder.
Revised: where F i − 1 is the intermediate feature map, which can be initialized as the positional embedding[12, 40], learned constant [20], or Fourier features [36, 19] for F 0 in the decoder.

Original: µ ( · ) and σ ( · )respectively denote the functions for calculating the mean and standard deviation of the activation.
Revised: µ ( · ) and σ ( · )respectively denote the functions for calculating the mean and standard deviation of the activation.

Original: There are many choices of the normalization.
Revised: There are many choices of the normalization.

Original: Following the baseline VQGAN [11], here we directlyuse the Group Normalization [43].
Revised: Following the baseline VQGAN [11], here we directlyuse the Group Normalization [43].

Original: ϕ γ ( · ) and ϕ β ( · ) are two learned affine transformations, which areimplemented as 1 × 1 convolutional filters in our setting, to convert the discrete representation
Revised: ϕ γ ( · ) and ϕ β ( · ) are two learned affine transformations, which areimplemented as 1 × 1 convolutional filters in our setting, to convert the discrete representation

Original: z q tothe scaling and bias values.
Revised: z q tothe scaling and bias values.

Original: Note that, the output of ϕ γ ( · ) and ϕ β ( · ) hold the same resolution to thecurrent activation F i − 1 , which injects spatial variances into the discrete feature.
Revised: Note that, the output of ϕ γ ( · ) and ϕ β ( · ) hold the same resolution to thecurrent activation F i − 1 , which injects spatial variances into the discrete feature.

Original: In fact, this spatially conditional normalization is derived from the adaptive instance normalization(AdaIN), which has been applied for many image synthesis approaches, such as style transfer [9, 17],image-to-image translation [18, 27], and image generation [20, 41].
Revised: In fact, this spatially conditional normalization is derived from the adaptive instance normalization(AdaIN), which has been applied for many image synthesis approaches.

Original: Compared to these methods, ourspatially conditional map is a quantized map, which contains learned compact contents.
Revised: Compared to these methods,our spatially conditional map is a quantized map, which contains learned compact contents.

Original: As suchmodulation operator injects the spatially variant information, it encourages the same quantizationentries to generate plausible and diverse results on different locations.
Revised: As suchmodulation operator injects the spatially variant information, it encourages the same quantizationentries to generate plausible and diverse results on different locations.

Original: Multichannel representation.
Revised: Multichannel representation.

Original: Unlike existing quantizers that map an image into a single-channelindex map, here we convert it into a multichannel index map with the shared codebook to furtherimprove the image quality, similar to [47].
Revised: Unlike existing quantizers that map an image into a single-channelindex map, here we convert it into a multichannel index map with the shared codebook to furtherimprove the image quality, similar to [47].

Original: This is inspired by the conventional GAN setting,where 1 × 1 × n c random vector is able to generate photorealistic images with reasonable structure[13, 20, 21], showing that values across the channels contain abundant information.
Revised: This is inspired by the conventional GAN setting,where 1 × 1 × n c random vector is able to generate photorealistic images with reasonable structure[13, 20, 21], showing that values across the channels contain abundant information.

Original: In practice,we first subdivide the encoded continuous feature ˆ z ∈ R h × w × n z along the channel dimensioninto multiple chunks, i.e. ˆ z
Revised: In practice,we first subdivide the encoded continuous feature ˆ z ∈ R h × w × n z along the channel dimensioninto multiple chunks, i.e. ˆ z

Original: =
Revised: =

Original: ˆ z (1) , ·
Revised: ˆ z (1) , ·

Original: · · , ˆ z ( c ) } , ˆ z ( c ) ∈ R h × w × n z /c .
Revised: · · , ˆ z ( c ) } , ˆ z ( c ) ∈ R h × w × n z /c .

Original: Each of these chunks is thenquantized based on the equation 2 to the closest codevectors in the codebook.
Revised: Each of these chunks is thenquantized based on the equation 2 to the closest codevectors in the codebook.

Original: For 256 × 256 images, we downsample it by a fixed factor of 16 to 16 × 16 features.
Revised: For 256 × 256 images, we downsample it by a fixed factor of 16 to 16 × 16 features.

Original: Unless otherwisenoted, in this work we employ c = 4 parallel chunks, and our equivalent sequence representationis 16 × 16 × 4 , which is 192x times smaller than the original image.
Revised: Unless otherwisenoted, in this work we employ c = 4 parallel chunks, and our equivalent sequence representationis 16 × 16 × 4 , which is 192x times smaller than the original image.

Original: Note that, in such way thecodevector dimensionality in the codebook will be reduced to n q = n z /c .
Revised: Note that, in such way thecodevector dimensionality in the codebook will be reduced to n q = n z /c .

Original: Due to this reduceddimension of each codevector, the total computational cost is similar to that of the single-channelrepresentation with fully dimensionality.
Revised: Due to this reduceddimension of each codevector, the total computational cost is similar to that of the single-channelrepresentation with fully dimensionality.

Original: In fact, our 
Revised: In fact, our codebook Z ∈ R K × n q has a smaller size withthe number of dimensions to be n q = n z /c = 256 / 4 = 64 .

Original: While a higher resolution representation, e.g. 32 × 32 × 1 or 64 × 64 × 1 , can also improve thereconstruction quality as in [30, 32, 45], the computational cost will be expensive for the secondstage due to the longer sequence s = h × w (see Sec. 3.2 for details).
Revised: While a higher resolution representation, e.g. 32 × 32 × 1 or 64 × 64 × 1 , can also improve thereconstruction quality as in [30, 32, 45], the computational cost will be expensive for the secondstage due to the longer sequence s = h × w (see Sec. 3.2 for details).

Original: In contrast, our multichannelrepresentation maintains a smaller sequence length, i.e. 16 × 16 , although each sequence tokennow is recomposed by four pieces, each of which corresponds to one quantization index.
Revised: In contrast, our multichannelrepresentation maintains a smaller sequence length, i.e. 16 × 16 , although each sequence tokennow is recomposed by four pieces, each of which corresponds to one quantization index.

Original: Withsuch an exponential combination capacity of K c for each spatial grid feature ˆ z ij , the multichannelrepresentation power becomes much larger than the K entries in the original codebook.
Revised: Withsuch an exponential combination capacity of K c for each spatial grid feature ˆ z ij , the multichannelrepresentation power becomes much larger than the K entries in the original codebook.

Original: 3.2 Modeling Prior Distribution
Revised: 3.2 Modeling Prior Distribution

Original: While a decoder can invert the discrete image embeddings z q to produce images x , we need to tame amodel to estimate the underlying prior distribution over the discrete space to enable image generation.
Revised: While a decoder can invert the discrete image embeddings z q to produce images x , we need to tame amodel to estimate the underlying prior distribution over the discrete space to enable image generation.

Original: Since our main goal is to improve the codebook learning stage, we directly employ the existingsetup for the second stage, including the conventional autoregressive model and the latest maskedgenerative image transformer (MaskGIT) [2].
Revised: Since our main goal is to improve the codebook learning stage, we directly employ the existingsetup for the second stage, including the conventional autoregressive model and the latest maskedgenerative image transformer (MaskGIT) [2].

Original: Autoregressive token generation.
Revised: Autoregressive token generation.

Original: After embedding an image x into an index sequence s ={ s 1 , · · · , s h × w } of the codebook entries z q , the image generation can be naturally formulated as anautoregressive next-symbol predication problem.
Revised: After embedding an image x into an index sequence s ={ s 1 , · · · , s h × w } of the codebook entries z q , the image generation can be naturally formulated as anautoregressive next-symbol predication problem.

Original: In particular, we maximize the likelihood function:
Revised: In particular, we maximize the likelihood function:

Original: [Equation]
Revised: [Equation]

Original: where p ( s i | s <i ) is the probability of having the next symbol s
Revised: where p ( s i | s <i ) is the probability of having the next symbol s

Original: i given all the previous symbols s <i .
Revised: i given all the previous symbols s <i .

Original: Then, the training objective for the second stage is equal to minimize the negative log-likelihood ofthe whole sequence, i.e. L = E x ∼ p ( x ) [ − log p ( s )] .
Revised: Then, the training objective for the second stage is equal to minimize the negative log-likelihood ofthe whole sequence, i.e. L = E x ∼ p ( x ) [ − log p ( s )] .

Original: There are many choices of autoregressive generation models, such as CNN-based PixelCNN [38]and transformer-based GPT [3].
Revised: There are many choices of autoregressive generation models, such as CNN-based PixelCNN [38]and transformer-based GPT [3].

Original: While they can sequentially produce diverse and reasonable resultsbased on the previously generated results, the sampling process is very slow.
Revised: While they can sequentially produce diverse and reasonable resultsbased on the previously generated results, the sampling process is very slow.

Original: Masked token generation.
Revised: Masked token generation.

Original: To produce the index sequence efficiently, we follow the latest MaskGITto simultaneously predict all tokens in parallel.
Revised: To produce the index sequence efficiently, we follow the latest MaskGITto simultaneously predict all tokens in parallel.

Original: Inspired by BERT [5], the model aims to estimate thedistribution of all indices based on the visible indices in a fixed length:
Revised: Inspired by BERT [5], the model aims to estimate thedistribution of all indices based on the visible indices in a fixed length:

Original: [Equation]
Revised: [Equation]

Original: where s ¯ m denotes conditional representation with partially visible codebook indices.
Revised: where s ¯ m denotes conditional representation with partially visible codebook indices.

Original: In practice,the masked map s ¯ m , with random mask ratios during training, is fed into a multi-layer bidirectionaltransformer to estimate the possible distribution of each masked indices, where the negative loglikelilood objective as above is also calculated between the ground-truth one-hot index and predictedindex.
Revised: In practice,the masked map s ¯ m , with random mask ratios during training, is fed into a multi-layer bidirectionaltransformer to estimate the possible distribution of each masked indices, where the negative loglikelilood objective as above is also calculated between the ground-truth one-hot index and predictedindex.

Original: At inference time, the model runs fixed steps and predicts all tokens simultaneously in parallelbased on the previous produced visible tokens at each iteration.
Revised: At inference time, the model runs fixed steps and predicts all tokens simultaneously in parallelbased on the previous produced visible tokens at each iteration.

Original: Then, the most confident tokens areselected out and the remaining masked tokens will be predicted in the next step condition on previouspredicted tokens.
Revised: Then, the most confident tokens areselected out and the remaining masked tokens will be predicted in the next step condition on previouspredicted tokens.

Original: We refer readers to MaskGIT [2] for more details.
Revised: We refer readers to MaskGIT [2] for more details.

Original: Multichannel token generation.
Revised: Multichannel token generation.

Original: Since our MoVQ embeds an image into a multichannel sequences = { s 1 , · · · , s h × w } , where s i ∈ { 0 , · · · , |Z| − 1 } c , for each position, we inversely concatenatechunks along the channel as one token for the input, and predict c indices for the output.
Revised: Since our MoVQ embeds an image into a multichannel sequences = { s 1 , · · · , s h × w } , where s i ∈ { 0 , · · · , |Z| − 1 } c , for each position, we inversely concatenatechunks along the channel as one token for the input, and predict c indices for the output.

Original: In practice,we independently mask each index for various channels and positions.
Revised: In practice,we independently mask each index for various channels and positions.

Original: Compared to MaskGIT, thepredicated indices in our model condition on three directional dependencies, i.e. visible indices fromdifferent positions and channels, resulting in larger diversity due to the combination capability.
Revised: Compared to MaskGIT, thepredicated indices in our model condition on three directional dependencies, i.e. visible indices fromdifferent positions and channels, resulting in larger diversity due to the combination capability.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Experimental Details
Revised: 4.1 Experimental Details

Original: Datasets.
Revised: Datasets.

Original: To evaluate the proposed method, we instantiated MoVQ on both unconditional andclass-conditional image generation tasks, with FFHQ [20] and ImageNet [33] respectively.
Revised: To evaluate the proposed method, we instantiated MoVQ on both unconditional andclass-conditional image generation tasks, with FFHQ [20] and ImageNet [33] respectively.

Original: Thetraining and validation setting followed the default one of our baseline model VQGAN [11].
Revised: Thetraining and validation setting followed the default one of our baseline model VQGAN [11].

Original: Wetrained all
Revised: Wetrained all

Original: [Table]
Revised: [Table]

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: For image generation, we used the two most common evaluation metrics
Revised: For image generation, we used the two most common evaluation metrics

Original: Fréchet Inception Distance ( FID ) [15] and Inception Score ( IS ) [34] to evaluate the quality anddiversity between generated images and ground truth images.
Revised: Fréchet Inception Distance ( FID ) [15] and Inception Score ( IS ) [34] to evaluate the quality anddiversity between generated images and ground truth images.

Original: Since we focused on the first stage,we also evaluated the quality between reconstructed images and original images.
Revised: Since we focused on the first stage,we also evaluated the quality between reconstructed images and original images.

Original: Except for therFID score, we additionally reported the traditional patch-level image quality metrics, including peaksignal-to-noise ratio (PSNR), structure similarity index (SSIM), and the latest learned feature-level
Revised: Except for therFID score, we additionally reported the traditional patch-level image quality metrics, including peaksignal-to-noise ratio (PSNR), structure similarity index (SSIM), and the latest learned feature-level

Original: LPIPS [46] for the paired reconstructed images and ground truth images.
Revised: LPIPS [46] for the paired reconstructed images and ground truth images.

Original: 4.2 Image Quantization
Revised: 4.2 Image Quantization

Original: Network structures and implementation details.
Revised: Network structures and implementation details.

Original: For each dataset, we only trained a singlescale quantizer with a codebook Z ∈ R 1024 × 64 , i.e. 1024 codevectors each with 64 dimensions, on256 × 256 images for all experiments.
Revised: For each dataset, we only trained a singlescale quantizer with a codebook Z ∈ R 1024 × 64 , i.e. 1024 codevectors each with 64 dimensions,on 256 × 256 images for all experiments.

Original: Our encoder-decode pipelinefor this first stage is built uponthe original VQGAN 1 , except the only difference that we replaced the original Group Normalizationwith the proposed spatially conditional normalization layer, with the multichannel representation(4-channel by default).
Revised: Our encoder-decode pipeline is built upon the original VQGAN 1 , except the only difference that we replaced the original Group Normalization with theproposed spatially conditional normalization layer.

Original: We applied the new normalization layer in the first three blocks of the decoder.
Revised: We applied the new normalization layer in thefirst three blocks of the decoder.

Original: Following the default setting in VQGAN, images are always downsampled by a fixed factor of 16, i.e.from 256 × 256 × 3 to a grid of tokens with the size of 16 × 16 × 4 , totally 256 tokens each with 4channels.
Revised: Following the default setting in VQGAN, images are alwaysdownsampled by a fixed factor of 16, i.e. from 256 × 256 × 3 to a grid of tokens with the size of16 × 16 × 4 .

Original: We set hyper-parameters following the baseline VQGAN work, and we trained all modelswith a batch size of 48 across 4 Tesla V100 GPUs with 40 epochs for this stage.
Revised: We set hyper-parameters following the baseline VQGAN work, and we trained allmodels with a batch size of 48 across 4 Tesla V100 GPUs with 40 epochs for this stage.

Original: We compare MoVQ with the state-of-the-art methods for image reconstruction in Table 1.
Revised: We compare MoVQ with the state-of-the-art methods for image reconstruction in Table 1.

Original: Allinstantiations of our model outperform the state-of-the-art methods under the same compressionratio (192x).
Revised: Allinstantiations of our model outperform the state-of-the-art methods under the same compressionratio (192x).

Original: This includes ViT-VQGAN [45] and RQ-VAE [24], which utilize higher resolution
Revised: This includes ViT-VQGAN [45] and RQ-VAE [24], which utilize higher resolution

Original: [Table]
Revised: [Table]

Original: representation and recursively residual representation, respectively.
Revised: representation and recursively residual representation, respectively.

Original: Without bells and whistles,though we use a much smaller number of parameters (82.7M), similar to VQGAN (72.1M), MoVQoutperformers ViT-VQGAN [45], which employs a larger transformer model (599M) on higherresolution representation ( 32 × 32 ) for the first stage.
Revised: Without bells and whistles,though we use a much smaller number of parameters (82.7M), similar to VQGAN (72.1M), MoVQoutperformers ViT-VQGAN [45], which employs a larger transformer model (599M) on higherresolution representation ( 32 × 32 ) for the first stage.

Original: The concurrent RQ-VAE work [24] alsorepresents an image with multiple channels by recursively calculating the residual informationbetween quantized vectors and their continuous ones, which requires much more embedding times.
Revised: The concurrent RQ-VAE work [24] alsorepresents an image with multiple channels by recursively calculating the residual informationbetween quantized vectors and their continuous ones, which requires much more embedding times.

Original: Furthermore, the entries in RQ-VAE are not equally important because the residual representationhighlights the codevectors in the first round, leaving the other codevectors to capture the small residualinformation.
Revised: Furthermore, the entries in RQ-VAE are not equally important because the residual representationhighlights the codevectors in the first round, leaving the other codevectors to capture the small residualinformation.

Original: In contrast, our codevectors share the same significance in all channels, resulting inlarger representation capability.
Revised: In contrast, our codevectors share the same significance in all channels, resulting inlarger representation capability.

Original: More importantly, our model dramatically improves the reconstructedimages quality on all metrics, suggesting the reconstructed images are closer to the original inputs,which contributes to more downstream image interpolation tasks.
Revised: More importantly, our model dramatically improves the reconstructedimages quality on all metrics, suggesting the reconstructed images are closer to the original inputs,which contributes to more downstream image interpolation tasks.

Original: Note that, our learned codebook contains only 1024 codevectors with 64 dimensionality, but interestingly outperforms other methods using larger codebook sizes.
Revised: Note that, our learned codebook contains only 1024 codevectors with 64 dimensionality, but interestingly outperforms other methods using larger codebook sizes.

Original: This suggests that a better quantizercan improve the codebook usage, and it is not necessary to greedily increase the codebook size.
Revised: This suggests that a better quantizercan improve the codebook usage, and it is not necessary to greedily increase the codebook size.

Original: Moredetails are discussed in Sec. 4.4.

Original: The qualitativereconstruction results are visualized in Figs.
Revised: The qualitative results are visualized in Figs.

Original: 3 and 4.
Revised: 3 and 4.

Original: MoVQ achieves impressiveresults under various conditions.
Revised: MoVQ achieves impressive results undervarious conditions.

Original: In Fig.
Revised: In Fig.

Original: 3, we compare our MoVQ with the baseline model VQGAN[11] and the concurrent model RQ-VAE[24].
Revised: 3, we compare our MoVQ with the baseline model VQGAN [11] and theconcurrent model RQ-VAE

Original: VQGANclearly holds repeated artifacts on the similarsemantic patches, such as the grass and trees.
Revised: VQGAN holds repeated artifacts on the similar semantic patches,such as the grass and trees.

Original: RQ-VAE improves the visual appearance, but exhibitssystematic artifacts with lossy information.
Revised: RQ-VAE improves the visual appearance, but exhibits systematic artifactswith lossy information.

Original: Our MoVQ shows no such artifacts, providing muchmore realistic details.
Revised: Our MoVQ shows no such artifacts, providing much more realistic details.

Original: 4.3 Image Generation
Revised: 4.3 Image Generation

Original: Network structures and implementation details.
Revised: Network structures and implementation details.

Original: After embedding images as a sequence withs = h × w = 16 × 16 = 256 tokens, a bidirectional transformer is implemented to estimate theunderlying prior distribution in the second stage.
Revised: After embedding images as a sequence, atransformer is implemented to estimate the underlying prior distribution in the second stage.

Original: Here, we used the same configuration for all models:24 layers, 16 attention heads, 1024 embedding dimensions and 4096 hidden dimensions.
Revised: Here, weused the same configuration for all models: 24 layers, 16 attention heads, 1024 embedding dimensionsand 4096 hidden dimensions.

Original: The networkarchitecture is also similar to the original VQGAN, except that we predicted a 16 × 16 × 4 × 1024tensor in the final layer, where 16 × 16 × 4 is the number of predicted indexes, and 1024 is thenumber of entries in our learned codebook.
Revised: The network architecture is build upon the VQGAN baseline, exceptthat we predicted a 16 × 16 × 4 × 1024 tensor in the final layer, where 16 × 16 × 4 is the number ofpredicted indexes, and 1024 is the number of entries in our learned codebook.

Original: Besides, unless otherwise noted, we predict all tokenssimultaneously based on the visible tokens as in [5, 2] during the training. The inference strategy isinspired by MaskGIT [2] 2 , which samples tokens  in a fixed number of steps, instead of autoregressivepredicting tokens one-by-one.
Revised: Here, we set the imagegeneration in two scenarios: (1) an " auto " scenario, in which the training and inference are totallysimilar with the VQGAN baseline, and (2) a " mask " scenario, in which the training and inferenceis inspired by MaskGIT [2] 2 . Unless otherwise noted, we default samples tokens as in MaskGITfor much faster sampling.

Original: The hyper-parameters follow the defaults setting as in VQGAN, andwe trained all models with a batch size of 64 across 4 Tesla V100 GPUs with 200 epochs.
Revised: The hyper-parameters follow the defaults setting as in VQGAN, and wetrained all models with a batch size of 64 across 4 Tesla V100 GPUs with 200 epochs.

Original: The unconditional generation results are compared in Table 2.
Revised: The unconditional generation results are compared in Table 2.

Original: Following MaskGIT, we do not set anyspecial sampling strategies such as top- k and top- p sampling heuristics.
Revised: Following MaskGIT, we do not set anyspecial sampling strategies such as top- k and top- p sampling heuristics.

Original: We produce 60,000 samplesat the inference time.
Revised: We produce 60,000 samplesat the inference time.

Original: Our model outperforms most VQ-based methods, with smaller parameters andfaster inference time.
Revised: Our model outperforms most VQ-based methods, with smaller parameters andfaster inference time.

Original: While our model performs worse than the concurrent work ViT-VQGAN, theyuse a much larger model (5 times than ours) in the second stage with a longer training, which is not a
Revised: While our model performs worse than the concurrent work ViT-VQGAN, theyuse a much larger model (5 times than ours) in the second stage with a longer training, which is not a

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: directly fair comparison.
Revised: directly fair comparison.

Original: A few generated samples are visualized in Figures 1 (see the Appendix formore).
Revised: A few generated samples are visualized in Figures 1 (see the Appendix formore).

Original: MoVQ generates photorealistic high-fidelity images, with a large diversity.
Revised: MoVQ generates photorealistic high-fidelity images, with a large diversity.

Original: We also compare the proposed MoVQ with state-of-the-art models for class-conditional imagegeneration on ImageNet 256 × 256 .
Revised: We also compare the proposed MoVQ with state-of-the-art models for class-conditional imagegeneration on ImageNet 256 × 256 .

Original: Our model generates 50 samples for each category, 50,000samples in total, for the quantitative evaluation.
Revised: Our model generates 50 samples for each category, 50,000samples in total, for the quantitative evaluation.

Original: As shown in Table 3, the proposed MoVQ significantlyimprove the performance over the baseline model VQGAN [11].
Revised: As shown in Table 3, the proposed MoVQ significantlyimprove the performance over the baseline model VQGAN [11].

Original: Compared with the concurrent work
Revised: Compared with the concurrent work

Original: RQ-VAE [24], which also employs a multi-channel representation, our performance is competitivewith them, even without the specially designed RQ-transformer for further predicting multichannelindexes.
Revised: RQ-VAE [24], which also employs a multi-channel representation, our performance is competitivewith them, even without the specially designed RQ-transformer for further predicting multichannelindexes.

Original: As MaskGIT and VIT-VQGAN use more GPUs for longer training, it is not a directly faircomparison with our MoVQ.
Revised: As MaskGIT and VIT-VQGAN use more GPUs for longer training, it is not a directlyfair comparison with our MoVQ.

Original: Figures 5 show a few generated samples (see Appendix for more).
Revised: Figures 5 show a few generated samples.

Original: Notethat, the proposed MoVQ appears to be able to generate high frequency details, such as the eyes ofthe bird.
Revised: Note that, the proposed MoVQ appears to be able to generate high frequency details, such as the eyes of the bird.

Original: 4.4 Discussion
Revised: 4.4 Discussion

Original: We run a few ablations to analyze the proposed MoVQ.
Revised: We run a few ablations to analyze the proposed MoVQ.

Original: The results are summarized in Fig.
Revised: The results are summarized in Fig.

Original: Architecture.
Revised: Architecture.

Original: Fig.
Revised: Fig.

Original: 6 (a) reports MoVQ with various decoder settings.
Revised: 6 (a) reports MoVQ with various decoder settings.

Original: Compared with the baselinemodel VQGAN ( A ), although we used the same settings for the network architecture, the multichannelrepresentation ( B ) naturally leads to a significant improvement by decomposing and recomposing thefeatures in channel.
Revised: Compared with the baselinemodel VQGAN ( A ), although we used the same settings for the network architecture, the multichannelrepresentation ( B ) naturally leads to a significant improvement by decomposing and recomposing thefeatures in channel.

Original: For the spatially conditional normalization ( C ), we explored three most commonlyused embedding functions as the initial feature F 0 , including positional embedding [12, 40], learnedconstant [20], and Fourier features [36, 19].
Revised: For the spatially conditional normalization ( C ), we explored three most commonlyused embedding functions as the initial feature F 0 , including positional embedding [12, 40], learnedconstant [20], and Fourier features [36, 19].

Original: For positional embedding, we implemented sine andcosine functions of different frequencies on different locations and channels.
Revised: For positional embedding, we implemented sine andcosine functions of different frequencies on different locations and channels.

Original: The learned constant isimplemented as learned parameters with the same size as the embedded discrete feature.
Revised: The learned constant isimplemented as learned parameters with the same size as the embedded discrete feature.

Original: However,the improvement from both methods is limited.
Revised: However,the improvement from both methods is limited.

Original: We believe a key reason is the lack of special identity
Revised: We believe a key reason is the lack of special identity

Original: Methods rFID ↓
Revised: Methods rFID FID

Original: A Baseline VQGAN 4.42  B + multichannel x 3.
Revised: A Baseline VQGAN 4.42 11.4 B + multichannel x 3.78 10.

Original: C w/ sinusiods 3.
Revised: C w/ sinusiods 3.52 9.

Original: w/ learned constants 3.
Revised: w/ learned constants 3.48 8.

Original: w/ Fourier features 2.
Revised: w/ Fourier features 2.26 8.

Original: Methods Num Z rFID↓
Revised: Methods Num Z rFID

Original: VQGAN6.
Revised: VQGAN6.

Original: VQGAN3.
Revised: VQGAN3.

Original: MoVQ 1. MoVQ 1.
Revised: MoVQ 1. MoVQ 1.

Original: 16 channels 1.
Revised: 16 channels 1.

Original: r F I D
Revised: r F I D

Original: Reconstruction FID Generation FID 8.8 8.
Revised: Reconstruction FID Generation FID 8.8 8.

Original: F I D
Revised: F I D

Original: [Figure]
Revised: [Figure]

Original: for spatiallyconditional normalization on FFHQ. [Figure] [Figure] geNet.
Revised: spatially

Original: [Figure]

Original: and generation

Original: [Figure]
Revised: [Figure]

Original: information for the reconstruction task, while they can provide high frequency signals.
Revised: information for the reconstruction task, while they can provide high frequency signals.

Original: In contrast,each Fourier feature is calculated from the corresponding discrete feature, which contains the identityfeature, as well as the high-frequency signals, resulting in a significant improvement.
Revised: In contrast,each Fourier feature is calculated from the corresponding discrete feature, which contains the identityfeature, as well as the high-frequency signals, resulting in a significant improvement.

Original: Codebook size.
Revised: Codebook size.

Original: Vanilla VQGAN shows a larger codebook leads to a much better representation.
Revised: Vanilla VQGAN shows a larger codebook leads to a much better representation.

Original: However, this is not true in our experimental results.
Revised: However, this is not true in our experimental results.

Original: In Fig.
Revised: In Fig.

Original: 6 (b) we compare the results withdifferent codebook sizes.
Revised: 6 (b) we compare the results withdifferent codebook sizes.

Original: As shown in the figure, the difference between the 1024 codebook entriesand 16384 entries in our setting is negligible.
Revised: As shown in the figure, the difference between the 1024 codebook entriesand 16384 entries in our setting is negligible.

Original: This suggests that once a better quantizer is designed, itis sufficient to represent vast amounts of images using a small codebook, which makes the modeleasier to train and evaluation.
Revised: This suggests that once a better quantizer is designed, itis sufficient to represent vast amounts of images using a small codebook, which makes the modeleasier to train and evaluation.

Original: Besides, we found a low codebook usage in current VQ models, whileour design can somehow improve the codebook usage.
Revised: Besides, we found a low codebook usage in current VQ models, whileour design can somehow improve the codebook usage.

Original: We plan to investigate this issue in futurework.

Original: Latent size.
Revised: Latent size.

Original: Vector quantization is a compression system, and we exploit the influence of differentcompress ratios by using different latent sizes.
Revised: Vector quantization is a compression system, and we exploit the influence of differentcompress ratios by using different latent sizes.

Original: In order not to dramatically increase the computationalcost, we mainly compare the performance of the latent representation with different numbers ofchannels in Fig.
Revised: In order not to dramatically increase the computationalcost, we mainly compare the performance of the latent representation with different numbers ofchannels in Fig.

Original: 6 (c).
Revised: 6 (c).

Original: The reconstruction performance is naturally benefits from more channels dueto the lower compression ratio.
Revised: The reconstruction performance is naturally benefits from more channels dueto the lower compression ratio.

Original: However, we note that not all configurations automatically benefitfrom more channels on image generation, as we need to predict more information in the secondstage.
Revised: However, we note that not all configurations automatically benefitfrom more channels on image generation, as we need to predict more information in the secondstage.

Original: Thus, we select a trade-off between the reconstruction and the generation.
Revised: Thus, we select a trade-off between the reconstruction and the generation.

Original: It is worth to notethat this is an initial step toward combining spatial vectors in the likelihood model with the channelinformation as in GAN for the generation task.
Revised: It is worth to notethat this is an initial step toward combining spatial vectors in the likelihood model with the channelinformation as in GAN for the generation task.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we introduced MoVQ, a new model that is simple, efficient yet effective for generatingdiverse and plausible images using a better quantizer with a powerful transformer to estimating theprior.
Revised: In this work, we introduced MoVQ, a new model that is simple, efficient yet effective for generatingdiverse and plausible images using a better quantizer with a powerful transformer to estimating theprior.

Original: Our encoder and decoder architectures are kept simple as in the baseline framework VQGAN,except that we incorporate a spatially conditional normalization and multichannel latent maps into the
Revised: Our encoder and decoder architectures are kept simple as in the baseline framework VQGAN,except that we incorporate a spatially conditional normalization and multichannel latent maps into the

Original: VQGAN method.
Revised: VQGAN method.

Original: Experimental results show that MoVQ significantly outperforms the state-of-the-art
Revised: Experimental results show that MoVQ significantly outperforms the state-of-the-art

Original: VQ models on image modeling, yet without increasing computational cost.
Revised: VQ models on image modeling, yet without increasing computational cost.

Original: With a better quantizer,we show that the fidelity of our unconditional samples and class-conditional samples are better thanthe existing methods.
Revised: With a better quantizer,we show that the fidelity of our unconditional samples and class-conditional samples are better thanthe existing methods.

Original: In the future, we would like to explore the semantic meaning of each entry in the learned codebook.
Revised: In the future, we would like to explore the semantic meaning of each entry in the learned codebook.

Original: As our model not only generates high fidelity image as the state-of-the-art GAN, but also providesmuch better reconstructed images, we are excited about the future of VQ-based models and plan toapply them to more image inversion, interpolation and translation tasks.
Revised: As our model not only generates high fidelity image as the state-of-the-art GAN, but also providesmuch better reconstructed images, we are excited about the future of VQ-based models and plan toapply them to more image inversion, interpolation and translation tasks.

Original: Limitation.
Revised: Limitation.

Original: Although our MoVQ dramatically improves the image representation quality than thestate-of-the-art under the same compression ratio, the model sometimes generates images with ahigh-frequency appearance, while the structure information is missing.
Revised: Although our MoVQ dramatically improves the image representation quality than thestate-of-the-art under the same compression ratio, the model sometimes generates images with ahigh-frequency appearance, while the structure information is missing.

Original: We believe this is partiallydue to the multichannel representation we employed in our model.
Revised: We believe this is partiallydue to the multichannel representation we employed in our model.

Original: Therefore, a better generationmodel for modeling multichannel indexes needs to be further studied.
Revised: Therefore, a better generationmodel for modeling multichannel indexes needs to be further studied.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent 
Revised: spent on

Original: on participant compensation?
Revised: [Table]

Original: [N/A]
Revised: [N/A]