Original: Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer
Revised: Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images fromfully reflecting global contexts.
Revised: Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images from fully reflecting global contexts.

Original: To address the issue, we propose an effective imagegeneration framework of Draft-and-Revise with Contextual RQ-transformer toconsider global contexts during the generation process.
Revised: To address the issue, we propose an effective image generation framework of Draft-and-Revise with Contextual RQ-transformer to consider global contexts during the generation process.

Original: As a generalized VQ-VAE,
Revised: As a generalized VQ-VAE,

Original: RQ-VAE first represents a high-resolution image as a sequence of discrete codestacks.
Revised: RQ-VAE first represents a high-resolution image as a sequence of discrete code stacks.

Original: After code stacks in the sequence are randomly masked, Contextual RQTransformer is trained to infill the masked code stacks based on the unmaskedcontexts of the image.
Revised: After code stacks in the sequence are randomly masked, Contextual RQTransformer is trained to infill the masked code stacks based on the unmasked contexts of the image.

Original: Then, we propose the two-phase decoding, Draft-and-Revise,for Contextual RQ-Transformer to generates an image, while fully exploiting theglobal contexts of the image during the generation process.
Revised: Then, we propose the two-phase decoding, Draft-and-Revise, for Contextual RQ-Transformer to generate an image, while fully exploiting the global contexts of the image during the generation process.

Original: Specifically.
Revised: Specifically.

Original: in thedraft phase, our model first focuses on generating diverse images despite ratherlow quality.
Revised: in the draft phase, our model first focuses on generating diverse images despite rather low quality.

Original: Then, in the revise phase, the model iteratively improves the quality ofimages, while preserving the global contexts of generated images.
Revised: Then, in the revise phase, the model iteratively improves the quality of images, while preserving the global contexts of generated images.

Original: In experiments,our method achieves state-of-the-art results on conditional image generation.
Revised: In experiments, our method achieves state-of-the-art results on conditional image generation.

Original: Wealso validate that the Draft-and-Revise decoding can achieve high performance byeffectively controlling the quality-diversity trade-off in image generation.
Revised: We also validate that the Draft-and-Revise decoding can achieve high performance by effectively controlling the quality-diversity trade-off in image generation.

Original:  Introduction
Revised:  Introduction

Original: Learning discrete representations of images enables autoregressive (AR) models to achieve promisingresults onhigh-resolution image generation.
Revised: Learning discrete representations of images enables autoregressive (AR) models to achieve promising results on image generation.

Original: Here, an image is encoded into a feature map, which isrepresented as a sequence of discrete codes [13, 33] or code stacks [23].
Revised: Here, an image is encoded into a feature map, which is represented as a sequence of discrete codes [13, 36] or code stacks [26].

Original: Then, an AR model generatesa sequence of codes in the raster scan order and decodes the codes into an image.
Revised: Then, an AR model generates a sequence of codes in the raster scan order and decodes the codes into an image.

Original: Consequently, ARmodels show high performance and scalability on large-scale datasets [13, 23, 26].
Revised: Consequently, AR models show high performance and scalability on large-scale datasets [13, 26, 29].

Original: Despite the promising results of AR models, we postulate that the ability of AR models is limited due tothe lack of considering global contexts in the generation process.
Revised: Despite the promising results of AR models, we postulate that the ability of AR models is limited due to the lack of considering global contexts in the generation process.

Original: Specifically, since AR models generateimages by sequentially predicting the next code andattending to only precedent codes generated, they neither exploit the later part of the generated image norconsider the global contexts during generation.
Revised: Specifically, since AR models generate images by sequentially predicting the next code and attending to only precedent codes generated, they neither exploit the later part of the generated image nor consider the global contexts during generation.

Original: Forexample, Figure 1 (middle) shows that an AR modelfails to generate a coherent image, when it is askedto inpaint the masked region of Figure 1 (left) with a school bus.
Revised: For example, Figure 1 (middle) shows that an AR model fails to generate a coherent image, when it is asked to inpaint the masked region of Figure 1 (left) with a school bus.

Original: Such a failure is due to the inabilityof AR models to refer to the context of traffic lane on the right side of the masked region.
Revised: Such a failure is due to the inability of AR models to refer to the context of traffic lane on the right side of the masked region.

Original: To address this issue, we propose an effective image generation framework, Draft-and-Revise , with acontextual transformer to exploit the global contexts of images.
Revised: To address this issue, we propose an effective image generation framework, Draft-and-Revise , with a contextual transformer to exploit the global contexts of images.

Original: Given a randomly masked image, thecontextual transformer is first trained to infill the masks by bidirectional self-attentions similarly to
Revised: Given a randomly masked image, the contextual transformer is first trained to infill the masks by bidirectional self-attentions similarly to

Original: BERT [8].
Revised: BERT [8].

Original: To fully leverage the contextual prediction in generation, we propose Draft-and-Revisedecoding which has two phases, draft and revise , imitating the image generation process of a humanexpert who draws a draft first and iteratively revises the draft to improve its quality.
Revised: To fully leverage the contextual prediction in generation, we propose Draft-and-Revise decoding which has two phases, draft and revise , imitating the image generation process of a human expert who draws a draft first and iteratively revises the draft to improve its quality.

Original: In the draft phase,the model first infills an empty image to generate a draft image with diverse contents despite therather low-quality.
Revised: In the draft phase, the model first infills an empty image to generate a draft image with diverse contents despite the rather low-quality.

Original: In the revise phase, the visual quality of the draft is iteratively improved, while theglobal contexts of the draft are preserved and exploited.
Revised: In the revise phase, the visual quality of the draft is iteratively improved, while the global contexts of the draft are preserved and exploited.

Original: Consequently, our Draft-and-Revise withcontextual transformer effectively generates high-quality images with diverse contents.
Revised: Consequently, our Draft-and-Revise with contextual transformer effectively generates high-quality images with diverse contents.

Original: We use residual-quantized VAE (RQ-VAE) [23] to implement our image generation framework, since
Revised: We use residual-quantized VAE (RQ-VAE) [26] to implement our image generation framework, since

Original: RQ-VAE generalizes vector-quantized VAE (VQ-VAE)
Revised: RQ-VAE generalizes vector-quantized VAE (VQ-VAE)

Original: [33] by representing an image as a sequenceof code stacks instead of a sequence of codes.
Revised: [36] by representing an image as a sequence of code stacks instead of a sequence of codes.

Original: Then, we propose Contextual RQ-Transformer as acontextual transformer for masked code stack modeling of RQ-VAE.
Revised: Then, we propose Contextual RQ-Transformer as a contextual transformer for masked code stack modeling of RQ-VAE.

Original: Specifically, given a sequenceof randomly masked code stacks, Contextual RQ-Transformer first uses a bidirectional transformer tocapture the global contexts of unmasked code stacks.
Revised: Specifically, given a sequence of randomly masked code stacks, Contextual RQ-Transformer first uses a bidirectional transformer to capture the global contexts of unmasked code stacks.

Original: Based on the global contexts, the masked codestacks are predicted in parallel, while the codes in each masked code stack are sequentially predicted.
Revised: Based on the global contexts, the masked code stacks are predicted in parallel, while the codes in each masked code stack are sequentially predicted.

Original: In experiments, our Draft-and-Revise framework with Contextual RQ-Transformer achieves state-ofthe-art results on conditional image generation and remarkable improvements on image inpainting.
Revised: In experiments, our Draft-and-Revise framework with Contextual RQ-Transformer achieves state-ofthe-art results on conditional image generation and remarkable improvements on image inpainting.

Original: Inaddition, we demonstrate that Draft-and-Revise decoding can effectively control the quality-diversitytrade-off in image generation to achieve high performance.
Revised: In addition, we demonstrate that Draft-and-Revise decoding can effectively control the quality-diversity trade-off in image generation to achieve high performance.

Original: The main contributions of this paper are summarized as follows.
Revised: The main contributions of this paper are summarized as follows.

Original: We propose an intuitive andpowerful framework, Contextual RQ-Transformer , for masked code stack modeling of RQ-VAEbased on a bidirectional transformer.
Revised: We propose an intuitive and powerful framework, Contextual RQ-Transformer , for masked code stack modeling of RQ-VAE based on a bidirectional transformer.

Original: We propose a novel two-phase decoding Draft-and-Revise ,for bidirectional transformers to fully exploit the global contexts during image generation andachieve state-of-the-art results on class- and text-conditional image generation benchmarks.
Revised: We propose a novel two-phase decoding, Draft-and-Revise , for bidirectional transformers to fully exploit the global contexts during image generation and achieve state-of-the-art results on class- and text-conditional image generation benchmarks.

Original: 3) Anextensive ablation study validates the effectiveness of Draft-and-Revise decoding on controlling thequality-diversity trade-off and its capability to generate high-quality images with diverse contents.
Revised: An extensive ablation study validates the effectiveness of Draft-and-Revise decoding on controlling the quality-diversity trade-off and its capability to generate high-quality images with diverse contents.

Original:  Related Work
Revised:  Related Work

Original: Discrete Representation for Image Generation By representing an image as a sequence of codes,
Revised: Discrete Representation for Image Generation By representing an image as a sequence of codes,

Original: VQ-VAE [33] becomes an important part for high-resolution image generation [6, 10, 15,23, 26, 33],but suffers from low quality of reconstructed images.
Revised: VQ-VAE [36] becomes an important part for high-resolution image generation [6, 10, 15, 26, 29, 36], but suffers from low quality of reconstructed images.

Original: However, VQGAN [13] significantly improvesthe perceptual quality of reconstructed images by adding the adversarial and perceptual losses into thetraining objective of VQ-VAE.
Revised: However, VQGAN [13] significantly improves the perceptual quality of reconstructed images by adding the adversarial and perceptual losses into the training objective of VQ-VAE.

Original: As a generalized approach of VQ-VAE and VQGAN, RQ-VAE [23]represents an image as a sequence of code stacks, which consists of ordered codes, and reduces thesequence length, while preserving the reconstruction quality.
Revised: As a generalized approach of VQ-VAE and VQGAN, RQ-VAE [26] represents an image as a sequence of code stacks, which consists of ordered codes, and reduces the sequence length, while preserving the reconstruction quality.

Original: Then, RQ-Transformer [23] achieveshigh performance with lower computational costs on generating high-resolution images.
Revised: Then, RQ-Transformer [26] achieves high performance with lower computational costs on generating high-resolution images.

Original: However, asan AR model of RQ-VAE, RQ-Transformer cannot capture the global contexts of generated images.
Revised: However, as an AR model of RQ-VAE, RQ-Transformer cannot capture the global contexts of generated images.

Original: Generation Tasks with Bidirectional Transformers To overcome the limitation of AR modelson unidirectional architecture, bidirectional transformers have been studied for generative tasks.
Revised: Generation Tasks with Bidirectional Transformers To overcome the limitation of AR models on unidirectional architecture, bidirectional transformers have been studied for generative tasks.

Original: After a bidirectional transformer is trained to infill a random mask as the masked token modeling of
Revised: After a bidirectional transformer is trained to infill a random mask as the masked token modeling of

Original: BERT [8], an effective decoding method has been proposed for bidirectional transformers to generatetexts [14, 30], images [6, 35], or videos [16].
Revised: BERT [8], an effective decoding method has been proposed for bidirectional transformers to generate texts [14, 33], images [6, 38], or videos [16].

Original: For image generation, M6-UFC [35] first incorporatesthe masked token modeling in the codes of VQGAN [13] for image generation.
Revised: For image generation, M6-UFC [38] first incorporates the masked token modeling in the codes of VQGAN [13] for image generation.

Original: MaskGIT [6]proposes the confidence-based mask generation to achieve high performance on ImageNet [7].
Revised: MaskGIT [6] proposes the confidence-based decoding to achieve high performance on ImageNet [7].

Original: Similar to M6-UFC [35] and MaskGIT [6], we also incorporate the masked token modeling of RQ-VAE in a bidirectional transformer for image generation.
Revised: Similar to M6-UFC [38] and MaskGIT [6], we also incorporate the masked token modeling of RQ-VAE in a bidirectional transformer for image generation.

Original: However, instead of the confidencebased decoding [6, 35], we propose a novel two-phase decoding for a bidirectional transformerto achieve state-of-the-art performance and validate its effectiveness.
Revised: However, instead of the confidence-based decoding [6, 38], we propose a novel two-phase decoding for a bidirectional transformer to achieve state-of-the-art performance and validate its effectiveness.

Original: Recently, discrete diffusionmodels [1, 4, 12, 15] also uses bidirectional transformers to generate an image, and the trainingwith an absorbing state [1] is also the same to infill randomly masked sequence [4, 6].
Revised: Recently, discrete diffusion models [1, 4, 12, 15] also uses [Figure] Tokenizing (RQ-VAE [Figure]

Original: However,different from the reverse process of diffusion models, our decoding method has explicit two phasesto generate high-quality images with diverse contents.
Revised: Random [Figure] RQ

Original: [Figure]
Revised: [Figure]

Original: RQ-VAE
Revised: Transformer

Original: [Figure]
Revised: [Figure]

Revised: bidirectional transformers to generate an image, and the training with an absorbing state [1] is also the same to infill randomly masked sequence [4, 6].

Revised: However, different from the reverse process of diffusion models, our decoding method has explicit two phases to generate high-quality images with diverse contents.

Original:  Draft-and-Revise Framework for Effective Image Generation
Revised:  Draft-and-Revise Framework for Effective Image Generation

Original: In this section, we propose our Draft-and-Revise framework for effective image generation usingbidirectional contexts of images.
Revised: In this section, we propose our Draft-and-Revise framework for effective image generation using bidirectional contexts of images.

Original: We first review RQ-VAE [23] as a generalization of VQ-VAE.
Revised: We first review RQ-VAE [26] as a generalization of VQ-VAE.

Original: Then,we propose Contextual RQ-Transformer which is trained to infill a randomly masked sequence ofcode stacks of RQ-VAE by understanding bidirectional contexts of unmasked parts in the sequence.
Revised: Then, we propose Contextual RQ-Transformer which is trained to infill a randomly masked sequence of code stacks of RQ-VAE by understanding bidirectional contexts of unmasked parts in the sequence.

Original: Lastly, we propose draft-and-revise decoding for a bidirectional transformer to effectively generatehigh-quality images exploiting global contexts of images.
Revised: Lastly, we propose draft-and-revise decoding for a bidirectional transformer to effectively generate high-quality images exploiting global contexts of images.

Original: Figure 2 provides the overview of ourproposed framework, including Contextual RQ-Transformer and Draft-and-Revise decoding.
Revised: Figure 2 provides the overview of our proposed framework, including Contextual RQ-Transformer and Draft-and-Revise decoding.

Original: 3.1 Residual-Quantized Variational Autoencoder (RQ-VAE)
Revised: 3.1 Residual-Quantized Variational Autoencoder (RQ-VAE)

Original: RQ-VAE [23] represents an image as a sequence of code stacks.
Revised: RQ-VAE [26] represents an image as a sequence of code stacks.

Original: Let a codebook C
Revised: Let a codebook C

Original: = { ( k, e ( k )) }
Revised: = { ( k, e ( k )) }

Original: k ∈
Revised: k ∈

Original: [ K ]include pairs of a code k and its code embedding e ( k ) ∈ R n z , where K = |C| is the codebook sizeand n z is the dimensionality of e ( k ) .
Revised: [ K ] include pairs of a code k and its code embedding e ( k ) ∈ R n z , where K = |C| is the codebook size and n z is the dimensionality of e ( k ) .

Original: Given a vector z ∈ R n z , Q ( z ; C ) is defined as the code of z : 
Revised: Given a vector z ∈ R n z , Q ( z ; C ) is defined as the code of z : Q ( z

Original: [Equation]
Revised: [Equation]

Original: Then, RQ with depth D represents a vector as a code stack which consists of D codes:
Revised: Then, RQ with depth D represents a vector as a code stack which consists of D codes:

Original: [Equation]
Revised: [Equation]

Original: where k d is the d -th code of z .
Revised: where k d is the d -th code of z .

Original: Specifically, RQ first initializes the 0 -th residual vector as r 

Revised: Specifically, RQ first initializes the 0 -th residual vector as r

Original: = z , andthen recursively discretizes a residual vector r d − 1 and computes the next residual vector r d as
Revised: = z , and then recursively discretizes a residual vector r d − 1 and computes the next residual vector r d as

Original: [Equation]
Revised: [Equation]

Original:  d [Equation]
Revised: for d ∈ [ D ] .

Original: Q [Equation] for d ∈ [ D ] . Finally, z is approximated by the sum of the D code embeddings ˆ z := (cid:80) Dd =1 e ( k d ) .
Revised: Finally, z is approximated by the sum of the D code embeddings ˆ z := (cid:80) Dd =1 e ( k d ) .

Original: Weremark that RQ is a generalized version of VQ, as RQ with D =1 is equivalent to VQ.
Revised: We remark that RQ is a generalized version of VQ, as RQ with D =1 is equivalent to VQ.

Original: For D > 1 ,
Revised: For D > 1 ,

Original: RQ conducts a finer approximation of z as the quantization errors are sequentially reduced as dincreases.
Revised: RQ conducts a finer approximation of z as the quantization errors are sequentially reduced as d increases.

Original: Here, the coarse-to-fine approximation ensures the D codes to be sequentially dependent.
Revised: Here, the coarse-to-fine approximation ensures the D codes to be sequentially dependent.

Original: RQ-VAE represents an image as a map of code stacks.
Revised: RQ-VAE represents an image as a map of code stacks.

Original: Specifically, a given image X is first convertedto a low-resolution feature map Z = E ( X ) ∈
Revised: Specifically, a given image X is first converted to a low-resolution feature map Z = E ( X ) ∈

Original: R H
Revised: R H

Original: × W × n z , and then each feature vector Z hw at spatialposition ( h, w ) is discretized into a code stack by RQ with depth D .
Revised: × W × n z , and then each feature vector Z hw at spatial position ( h, w ) is discretized into a code stack by RQ with depth D .

Original: As a result, we get a map ofcode stacks S ∈ [ K ] H × W × D .
Revised: As a result, we get a map of code stacks S ∈ [ K ] H × W × D .

Original: Further details of RQ-VAE are referred to Appendix.
Revised: Further details of RQ-VAE are referred to Appendix.

Original: 3.2 Contextual Transformer for Image Generation with Global Contexts
Revised: 3.2 Contextual Transformer for Image Generation with Global Contexts

Original: As a bidirectional transformer for RQ-VAE, we propose Contextual RQ-Transformer for imagegeneration based on a contextual understanding of images.
Revised: As a bidirectional transformer for RQ-VAE, we propose Contextual RQ-Transformer for image generation based on a contextual understanding of images.

Original: First, we adopt the pretraining of BERT [8]to formulate a masked code stack modeling of RQ-VAE.
Revised: First, we adopt the pretraining of BERT [8] to formulate a masked code stack modeling of RQ-VAE.

Original: Then, we introduce how Contextual RQTransformer infills the randomly masked code stacks after reading the given contextual information.
Revised: Then, we introduce how Contextual RQTransformer infills the randomly masked code stacks after reading the given contextual information.

Original: 3.2.1 Masked Code Stack Modeling of RQ-VAE
Revised: 3.2.1 Masked Code Stack Modeling of RQ-VAE

Original: By adopting the pretraining of BERT [8], we formulate the masked code stack modeling of RQ-VAEwith a contextual transformer to generate an image by iterative mask-infilling as non-AR models [14].
Revised: By adopting the pretraining of BERT [8], we formulate the masked code stack modeling of RQ-VAE with a contextual transformer to generate an image by iterative mask-infilling as non-AR models [14].

Original: We first convert the map S ∈
Revised: We first convert the map S ∈

Original: [ K ] H × W × D into a sequence of code stacks S ′ ∈
Revised: [ K ] H × W × D into a sequence of code stacks S ′ ∈

Original: [ K ] N × D using theraster-scan ordering, where N = HW and S ′ n = ( S ′ n 1 , · · · , S ′ nD ) ∈
Revised: [ K ] N × D using the raster-scan ordering, where N = HW and S ′ n = ( S ′ n 1 , · · · , S ′ nD ) ∈

Original: [ K ] D for n ∈ [ N ] .
Revised: [ K ] D for n ∈ [ N ] .

Original: We denote
Revised: We denote

Original: S ′ as S for the brevity of notation.
Revised: S ′ as S for the brevity of notation.

Original: A mask vector m is defined as a binary vector m ∈ { 0 , 1 } N toindicate the spatial positions to be masked.
Revised: A mask vector m is defined as a binary vector m ∈ { 0 , 1 } N to indicate the spatial positions to be masked.

Original: Then, the masked sequence S \ m of S by m is defined as
Revised: Then, the masked sequence S \ m of S by m is defined as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where [ MASK ] is a mask token to substitute for S nd if m n = 1 .
Revised: where [ MASK ] is a mask token to substitute for S nd if m n = 1 .

Original: Given a random mask vectorm ∼ q ( m ) , the masked code stacks given S \ m are modeled as
Revised: Given a random mask vector m ∼ q ( m ) , the masked code stacks given S \ m are modeled as

Original: [Equation]
Revised: [Equation]

Original: where q ( m ) is a mask distribution where the masking portion (cid:80) Nn
Revised: where q ( m ) is a mask distribution where the masking portion (cid:80) Nn

Original: =

Revised: =

Original: m i /N in (0 , 1] as well as themasking positions are randomly chosen.
Revised: m i /N in (0 , 1] as well as the masking positions are randomly chosen.

Original: Instead of fixing the portion to 15% as in BERT, training amodel with a random masking portion from (0 , 1] enables the model to generate new images based onvarious masking patterns including m n = 1 for all n .
Revised: Instead of fixing the portion to 15% as in BERT, training a model with a random masking portion from (0 , 1] enables the model to generate new images based on various masking patterns including m n = 1 for all n .

Original: We explain the details of q ( m ) in Section 3.2.3.
Revised: We explain the details of q ( m ) in Section 3.2.3.

Original: The left-hand side of Eq.
Revised: The left-hand side of Eq.

Original: 5 implies that all masked code stacks can be decoded in parallel based onthe contexts of S \ m .
Revised: 5 implies that all masked code stacks can be decoded in parallel based on the contexts of S \ m .

Original: If D = 1 , Eq. 5 becomes equivalent to conventional masked token modeling oftexts [8] and images [6, 16] where a single token at each masked position is predicted.
Revised: If D = 1 , Eq. 5 becomes equivalent to conventional masked token modeling of texts [8] and images [6, 16] where a single token at each masked position is predicted.

Original: For D > 1 , the
Revised: For D > 1 , the

Original: D codes of S n are autoregressively predicted, as they are sequentially computed in Eq.
Revised: D codes of S n are autoregressively predicted, as they are sequentially computed in Eq.

Original: 3 for a coarseto-fine approximation and hence well-suited for an AR prediction.
Revised: 3 for a coarseto-fine approximation and hence well-suited for an AR prediction.

Original: We show that the effectiveness ofour framework is generalizable regardless of D in Appendix B.
Revised: We show that the effectiveness of our framework is generalizable regardless of D in Appendix B.

Original: In addition, the parallel decoding cancontrol the trade-off between the quality and sampling speed of image generation.
Revised: In addition, the parallel decoding can control the trade-off between the quality and sampling speed of image generation.

Original: 3.2.2 Contextual RQ-Transformer
Revised: 3.2.2 Contextual RQ-Transformer

Original: We modify the previous RQ-Transformer [23] for masked code stack modeling with bidirectionalcontexts in Eq.
Revised: We modify the previous RQ-Transformer [26] for masked code stack modeling with bidirectional contexts in Eq.

Original: Contextual RQ-Transformer consists of Bidirectional Spatial Transformer and
Revised: Contextual RQ-Transformer consists of Bidirectional Spatial Transformer and

Original: Depth Transformer : Bidirectional Spatial Transformer understands contextual information in theunmasked code stacks using bidirectional self-attentions, and Depth Transformer infills the maskedcode stacks in parallel, by autoregressively predicting the D codes at each position.
Revised: Depth Transformer : Bidirectional Spatial Transformer understands contextual information in the unmasked code stacks using bidirectional self-attentions, and Depth Transformer infills the masked code stacks in parallel, by autoregressively predicting the D codes at each position.

Original: Bidirectional Spatial Transformer Given a masked sequence of code stacks S \ m , bidirectionalspatial transformer first embeds the masked sequence S \ m using the code embeddings of RQ-VAE as
Revised: Bidirectional Spatial Transformer Given a masked sequence of code stacks S \ m , bidirectional spatial transformer first embeds the masked sequence S \ m using the code embeddings of RQ-VAE as

Original: [Equation]
Revised: [Equation]

Original: where PE N ( n ) is an embedding for position n , and e [ MASK ] ∈ R n z is a mask embedding.
Revised: where PE N ( n ) is an embedding for position n , and e [ MASK ] ∈ R n z is a mask embedding.

Original: Then, thebidirectional self-attention blocks, f spatial θ , extracts the context vector h n to predict a code stack S n ,[Equation] [Equation]
Revised: Then, the bidirectional self-attention blocks, f spatial θ , extracts the context vector h n to predict a code stack S n ,

Original: [Equation]
Revised: [Equation]

Original: Depth Transformer
Revised: Depth Transformer

Original: The code stack at a masked position S n = ( S n 1 , · · · , S nD ) is autoregressivelypredicted.
Revised: The code stack at a masked position S n = ( S n 1 , · · · , S nD ) is autoregressively predicted.

Original: The input of depth transformer ( v nd )
Revised: The input of depth transformer ( v nd )

Original: Dd =1 is defined as
Revised: Dd =1 is defined as

Original: [Equation]
Revised: [Equation]

Original: where PE D ( d ) is the positional embedding for depth d . Then, depth transformer f depth θ , whichconsists ofcausal attention blocks, outputs the logit p nd to predict the d -th code  S nd at position n asp nd = [Equation] depth θ 
Revised: Require: A sequence of code stacks S , a partition Π = ( m 1 , · · · , m T ) , a model θ 1: for t = 1 , · · · , T do 2:

Original: [Equation]

Original: [Equation]

Original: v nd ) and p θ [Equation] S nd [Equation]
Revised: Sample S n ∼ p θ ( S n | S \ m t ) ∀ n : m tn =

Original: We remark that the architecture of Contextual RQ-Transformer subsumes bidirectional transformers.

Original: Specifically, RQ-Transformer with the depth D = 1 is equivalent to a bidirectional transformer sincethe depth transformer becomes a multilayer perceptron with layer normalization [2].

Original: Require: A sequence of code stacks S , a partition Π = ( m 1 , · · · , m T ) , a model θ 1: for t = 1 , · · · , T do 2:

Original: Sample S n ∼ p θ ( S n | S \ m t ) ∀ n : m tn =

Original: ▷ update the codes at masked positions 3: end for 4: return S
Revised: ▷ update the codes at masked positions 3: end for 4: return S

Original: Require: Partition sampling distributions p draft and p rev , the number of revision iterations M /* draft phase */ 1: S empty ← ([ MASK ] , · · · , [ MASK ]) N ▷ initialize empty code map 2: Sample Π ∼ p ( Π ; T draft ) 3: S draft ← UPDATE ( S empty , Π ; θ ) ▷ generate a draft code map /* revision phase */ 4: S 0 ← S draft 5: for m = 1 , · · · , M do 6:
Revised: Require: Partition sampling distributions p draft and p rev , the number of revision iterations M /* draft phase */ 1: S empty ← ([ MASK ] , · · · , [ MASK ]) N ▷ initialize empty code map 2: Sample Π ∼ p ( Π ; T draft ) 3: S draft ← UPDATE ( S empty , Π ; θ ) ▷ generate a draft code map /* revision phase */ 4: S 0 ← S draft 5: for m = 1 , · · · , M do 6:

Original: Sample Π ∼ p ( Π ; T revise ) 7:
Revised: Sample Π ∼ p ( Π ; T revise ) 7:

Original: S m ← UPDATE ( S m − 1 , Π ; θ ) ▷ iteratively revise the code map 8: end for 9: return S M 
Revised: S m ← UPDATE ( S m − 1 , Π ; θ ) ▷ iteratively revise the code map 8: end for 9: return S M where PE D ( d ) is the positional embedding for depth d . Then, depth transformer f depth θ , which consists of causal attention blocks, outputs the logit p nd to predict the d -th code S nd at position n as p nd = f depth θ ( v n 1 , · · · , v nd ) and p θ ( S nd = k | S n,<d , S \ m ) = softmax( p nd ) k .

Revised: We remark that the architecture of Contextual RQ-Transformer subsumes bidirectional transformers.

Revised: Specifically, RQ-Transformer with the depth D = 1 is equivalent to a bidirectional transformer since the depth transformer becomes a multilayer perceptron with layer normalization [2].

Original: 3.2.3 Training of Contextual RQ-Transformer
Revised: 3.2.3 Training of Contextual RQ-Transformer

Original: For the training of Contextual RQ-Transformer, let us define a mask distribution q ( m ) with a maskscheduling function γ .
Revised: For the training of Contextual RQ-Transformer, let us define a mask distribution q ( m ) with a mask scheduling function γ .

Original: Following previous approaches [6, 14, 16], the scheduling function γ is chosento be decreasing and to satisfy γ (0)
Revised: Following previous approaches [6, 14, 16], the scheduling function γ is chosen to be decreasing and to satisfy γ (0)

Original: = 1 and γ (1) = 0 .
Revised: = 1 and γ (1) = 0 .

Revised: Then, a mask m ∼ q ( m ) is specified as r ∼ Unif

Original: Then, a mask m ∼ q ( m ) is specified asr ∼ Unif([0 , 1)) and m ∼ Unif( { m : | m | = ⌈ γ ( r ) · N ⌉} ) , (10) where | m | = (cid:80) n ∈ [ N ] m n is the count of masked positions.
Revised: [Equation] )) and m ∼ Unif( { [Equation] where | m | = (cid:80) n ∈ [ N ] m n is the count of masked positions.

Original: Finally, the training objective of
Revised: Finally, the training objective of

Original: Contextual RQ-Transformer is to minimize the negative log-likelihood of masked code stacks:[Equation] n
Revised: Contextual RQ-Transformer is to minimize the negative log-likelihood of masked code stacks:

Original: [Equation]
Revised: [Equation]

Original: 3.3 Draft-and-Revise: Two-Phase Decoding with Global Contexts of Generated Imaegs
Revised: 3.3 Draft-and-Revise: Two-Phase Decoding with Global Contexts of Generated Images

Original: We propose a novel decoding algorithm, Draft-and-Revise , for Contextual RQ-Transformer toeffectively generate high-quality images with diverse visual contents.
Revised: We propose a novel decoding algorithm, Draft-and-Revise , for Contextual RQ-Transformer to effectively generate high-quality images with diverse visual contents.

Original: We remark that our effectivedecoding method is required for bidirectional transformers to fully exploit the global contexts ofimages and achieve high performance of image generation.
Revised: We remark that our effective decoding method is required for bidirectional transformers to fully exploit the global contexts of images and achieve high performance of image generation.

Original: Here, we introduce the details of our
Revised: Here, we introduce the details of our

Original: Draft-and-Revise decoding and then explain how the two-phase decoding of Draft-and-Revise caneffectively control the quality-diversity trade-off of generated images.
Revised: Draft-and-Revise decoding and then explain how the two-phase decoding of Draft-and-Revise can effectively control the quality-diversity trade-off of generated images.

Original: We define a partition Π
Revised: We define a partition Π

Original: =
Revised: =

Original: ( m 1 , · · · , m T ) as a collection of pairwise disjoint T mask vectors tocover all spatial positions, where (cid:80) Tt =1 m tn = 1 for all n ∈ [ N ] .
Revised: ( m 1 , · · · , m T ) as a collection of pairwise disjoint T mask vectors to cover all spatial positions, where (cid:80) Tt =1 m tn = 1 for all n ∈ [ N ] .

Original: A partition Π is sampled from thedistribution p ( Π ; T ) , which is the uniform distribution over all balanced partitions with size T : 
Revised: A partition Π is sampled from the distribution p ( Π ; T ) , which is the uniform distribution over all balanced partitions with size T : p [Equation] (cid:0)

Original: [Equation]
Revised: [Equation]

Original:  We first define a procedure UPDATE ( S , Π ) to update the sequence S as described in Algorithm 1,which updates S n with m tn = 1 for t ∈ 
Revised: [Equation] [Equation] [Table] We first define a procedure UPDATE ( S , Π ) to update the sequence S as described in Algorithm 1, which updates S n with m tn = 1 for t ∈ [ T ] .

Original: [ T ] .

Original: Then, Draft-and-Revise decoding in Algorithm 2generates a draft from the empty sequence of code stacks and improves the quality of the draft.
Revised: Then, Draft-and-Revise decoding in Algorithmgenerates a draft from the empty sequence of code stacks and improves the quality of the draft.

Original: Draft phase In the draft phase, our model gradually infills the empty sequence of code stacks togenerate a draft image, considering the global contexts of infilled code stacks.
Revised: Draft phase In the draft phase, our model gradually infills the empty sequence of code stacks to generate a draft image, considering the global contexts of infilled code stacks.

Original: Let S empty be an emptysequence of code stacks with S empty n =
Revised: Let S empty be an empty sequence of code stacks with S empty n =

Original: [ MASK ] D for all n .
Revised: [ MASK ] D for all n .

Original: Given a partition size T draft , our modelgenerates a draft image as
Revised: Given a partition size T draft , our model generates a draft image as

Original: S draft = UPDATE ( S empty , Π ; θ ) where Π ∼ p

Original: ( Π ; T draft ) .

Original: [Table]
Revised: [Equation]

Original: Revise phase The generated draft S draft is repeatedly revised to improve the visual quality of theimage, while preserving the overall structure of the draft.
Revised: Revise phase The generated draft S draft is repeatedly revised to improve the visual quality of the image, while preserving the overall structure of the draft.

Original: Given a partition size T revise and the numberof updates M , the draft S 

Revised: Given a partition size T revise and the number of updates M , the draft S

Original: =
Revised: =

Original: S draft is repeatedly updated M times as
Revised: S draft is repeatedly updated M times as

Original: S m = UPDATE ( S m − 1 , Π ; θ ) where Π ∼ p
Revised: [Equation] Π ∼ p

Original: ( Π ; T revise ) for m = 1 , · · · , M .
Revised: ( Π [Equation] T revise ) for m = 1 , · · · , M [Equation]

Original: Note that Draft-and-Revise is not a tailored method, since we can adopt any mask-infilling-basedgeneration method [4, 6] for UPDATE in Algorithm 1.
Revised: Note that Draft-and-Revise is not a tailored method, since we can adopt any mask-infilling-based generation method [4, 6] for UPDATE in Algorithm 1.

Original: For example, confidence-based decoding [6,16], which iteratively updates S from high-confidence to low-confidence predictions, can be used for
Revised: For example, confidence-based decoding [6, 16], which iteratively updates S from high-confidence to low-confidence predictions, can be used for

Original: UPDATE.
Revised: UPDATE.

Original: However, we find that confidence-based decoding generates low-diversity images withoversimplified contents, since a model tends to predict simple visual patterns with high confidence.
Revised: However, we find that confidence-based decoding generates low-diversity images with oversimplified contents, since a model tends to predict simple visual patterns with high confidence.

Original: Inaddition, confidence-based decoding often leads to biased unmasking patterns, which are not used intraining.
Revised: In addition, confidence-based decoding often leads to biased unmasking patterns, which are not used in training.

Original: We attach the detailed discussion about confidence-based decoding in Appendix C.
Revised: We attach the detailed discussion about confidence-based decoding in Appendix C.

Original: Thus, weuse a uniformly random partition Π in UPDATE as the most simplified rule, leaving investigationson sophisticated update methods as future work.
Revised: Thus, we use a uniformly random partition Π in UPDATE as the most simplified rule, leaving investigations on sophisticated update methods as future work.

Original: We postulate that our Draft-and-Revise can generate high-quality images with diverse contents byexplicitly dividing two phases.
Revised: We postulate that our Draft-and-Revise can generate high-quality images with diverse contents by explicitly dividing two phases.

Original: Specifically, a model first generates draft images with diverse visualcontents despite the rather low quality of drafts.
Revised: Specifically, a model first generates draft images with diverse visual contents despite the rather low quality of drafts.

Original: After semantically diverse images are generated asdrafts, we use sampling strategies such as temperature scaling [19] and classifier-free guidance [20]in the revise phase to improve the visual quality of the drafts, while preserving the major semanticcontents in drafts.
Revised: After semantically diverse images are generated as drafts, we use sampling strategies such as temperature scaling [19] and classifier-free guidance [20] in the revise phase to improve the visual quality of the drafts, while preserving the major semantic contents in drafts.

Original: Thus, our method can improve the performance of image generation by effectivelycontrolling the quality-diversity trade-off.
Revised: Thus, our method can improve the performance of image generation by effectively controlling the quality-diversity trade-off.

Original: In addition, we emphasize that the two-phased decoding isintuitive and resembles the image generation process of human experts, who repeatedly refine theirworks to improve the quality after determining the overall contents first.
Revised: In addition, we emphasize that the two-phased decoding is intuitive and resembles the image generation process of human experts, who repeatedly refine their works to improve the quality after determining the overall contents first.

Original:  Experiments
Revised:  Experiments

Original: In this section, we show that our Draft-and-Revise with Contextual RQ-Transformer can outperformprevious approaches for class- and text-conditional image generation.
Revised: In this section, we show that our Draft-and-Revise with Contextual RQ-Transformer can outperform previous approaches for class- and text-conditional image generation.

Original: In addition, we conduct anextensive ablation study to understand the effects of Draft-and-Revise decoding on the quality anddiversity of generated images, and the sampling speed.
Revised: In addition, we conduct an extensive ablation study to understand the effects of Draft-and-Revise decoding on the quality and diversity of generated images, and the sampling speed.

Original: We use the publicly released RQ-VAE [23]to represent a 256 × 256 resolution of images as 8 × 8 × 4 codes.
Revised: We use the publicly released RQ-VAE [26] to represent a 256 × 256 resolution of images as 8 × 8 × 4 codes.

Original: For a fair comparison, we make
Revised: For a fair comparison, we make

Original: Contextual RQ-Transformer have the same model size as the previous RQ-Transformer [23].
Revised: Contextual RQ-Transformer have the same model size as the previous RQ-Transformer [26].

Original: Fortraining, the quarter-period of cosine is used as the mask scheduling function γ in Eq.
Revised: For training, the quarter-period of cosine is used as the mask scheduling function γ in Eq.

Original: 10 followingthe previous studies [6, 24].
Revised: 10 following the previous studies [6, 27].

Original: We include the implementation details in Appendix.
Revised: We include the implementation details in Appendix.

Original: 4.1 Class-conditional Image Generation
Revised: 4.1 Class-conditional Image Generation

Original: We train Contextual RQ-Transformer with 333M, 821M, and 1.4B parameters on ImageNet [7] forclass-conditional image generation.
Revised: We train Contextual RQ-Transformer with 333M, 821M, and 1.4B parameters on ImageNet [7] for class-conditional image generation.

Original: For Draft-and-Revise decoding, we use T draft = 64 , T revise = 2 ,and M = 2 .
Revised: For Draft-and-Revise decoding, we use T draft = 64 , T revise = 2 , and M = 2 .

Original: We use temperature scaling [19] and classifier-free guidance [20] only in the revise phase,while none of the strategies are applied in the draft phase.
Revised: We use temperature scaling [19] and classifier-free guidance [20] only in the revise phase, while none of the strategies are applied in the draft phase.

Original: Fréchet Inception Distance (FID) [18],
Revised: Fréchet Inception Distance (FID) [18],

Original: Inception Score (IS) [28], and Precision and Recall [22] are used for evaluation measures.
Revised: Inception Score (IS) [31], and Precision and Recall [25] are used for evaluation measures.

Original: Table 1 shows that Contextual RQ-Transformer significantly outperforms the previous approaches.
Revised: Table 1 shows that Contextual RQ-Transformer significantly outperforms the previous approaches.

Original: Notably, Contextual RQ-Transformer with 333M parameters outperforms RQ-Transformers with1.4B and 3.8B parameters on all evaluation measures, despite having only about 4.2 × and 11.4 ×fewer parameters.
Revised: Notably, Contextual RQ-Transformer with 333M parameters outperforms RQ-Transformers with 1.4B and 3.8B parameters on all evaluation measures, despite having only about 4.2 × and 11.4 × fewer parameters.

Original: In addition, the performance is improved as the number of parameters increases to821M and 1.4B. Our model with 333M parameters is competitive with MaskGIT [6], and hence weconduct additional analysis on the effect of confidence-based decoding in Appendix C. Contextual
Revised: In addition, the performance is improved as the number of parameters increases to 821M and 1.4B. Our model with 333M parameters is competitive with MaskGIT [6], and hence we conduct additional analysis on the effect of confidence-based decoding in Appendix C. Contextual

Original: RQ-Transformer can achieve the lower FID score without a pretrained classifier than ADM-G and3.8B parameters of RQ-Transformer with the use of pretrained classifier.
Revised: RQ-Transformer can achieve the lower FID score without a pretrained classifier than ADM-G and 3.8B parameters of RQ-Transformer with the use of pretrained classifier.

Original: StyleGAN-XL also usesa pretrained classifier during both training and image generation and achieves the lowest FID in
Revised: StyleGAN-XL also uses a pretrained classifier during both training and image generation and achieves the lowest FID in

Original: Table 1.
Revised: Table 1.

Original: However, our model with 1.4B parameters has higher precision and recall than StyleGANXL, implying that our model generates images of better fidelity and diversity without a pretrainedclassifier.
Revised: However, our model with 1.4B parameters has higher precision and recall than StyleGANXL, implying that our model generates images of better fidelity and diversity without a pretrained classifier.

Original: Our high performance without a classifier is remarkable, since the gradient guidanceand rejection sampling are the tailored techniques to the model-based evaluation metrics in Table 1.
Revised: Our high performance without a classifier is remarkable, since the gradient guidance and rejection sampling are the tailored techniques to the model-based evaluation metrics in Table 1.

Original: Considering that the performance is marginally improved as the number of parameters increases from821M to 1.4B, an improved RQ-VAE can boost the performance of Contextual RQ-Transformer,since the reconstruction quality determines the best results of generated images.
Revised: Considering that the performance is marginally improved as the number of parameters increases from 821M to 1.4B, an improved RQ-VAE can boost the performance of Contextual RQ-Transformer, since the reconstruction quality determines the best results of generated images.

Original: 4.2 Text-conditional Image Generation
Revised: 4.2 Text-conditional Image Generation

Original: [Table] We train our model with 333M and 654Mparameters on CC-3M [32] for text-to-image(T2I) generation as described in Appendix A. 
Revised: We train our model with 333M and 654M parameters on CC-3M [35] for text-to-image (T2I) generation as described in Appendix A. A text condition is encoded into 32 tokens using Byte Pair

Original: We use Byte Pair Encoding [31, 34] to encodea text condition into 32 tokens.
Revised: Encoding [34, 37].

Original: Wealso report CLIP-score [25] with ViT-B/32 [11] as thecorrespondence between texts and images. 
Revised: We report CLIP-score [28] with ViT-B/32 [11] as the correspondence between texts and images. [Figure] [Figure] F I D

Revised: [Figure]

Original: Contextual RQ-Transformer in Table 2 outperforms the previous T2I generation models.
Revised: [Figure] i [Figure] i

Revised: [Figure]

Original: Contextual RQ-Transformer with 333M parameters achieves better FID than RQ-Transformerwith 654M parameters, and outperforms ImageBART and LDM-4, although our model has 12 × fewer parameters than ImageBART.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: When weincrease the number of parameters to 654M, our model achieves state-of-the-art FID on CC-3M. In
Revised: [Figure]

Original: Figure 3, our model generates images with unseen texts in CC-3M. T draft
Revised: [Figure]

Original: F I D
Revised: [Figure]

Original: [-Draft only Draft + Revise (a) 0 
 Number of revisions ( M ) 0.-]
Revised: [-Draft only Draft + Revise (a) 0 
 Number of revisions ( M ) 0.-][Table]

Original: P r e c i s i o n 0. R e c a ll Precision Recall (b)     64 T revise 2.
Revised: Contextual RQ-Transformer in Table 2 outperforms the previous T2I generation models.

Original: F I D Draft only Revise, M =1 Revise, M = (c)
Revised: Contextual RQ-Transformer with 333M parameters achieves better FID than RQ-Transformer with 654M parameters, and outperforms ImageBART and LDM-4, although our model has 12 × fewer parameters than ImageBART. When we increase the number of parameters to 654M, our model achieves state-of-the-art FID on CC3M. In Figure 3, our model generates images with unseen texts in CC-3M.

Original: 4.3 Conditional Image Inpainting
Revised: 4.3 Conditional Image Inpainting

Original: We conduct conditional image inpainting where a model infills a masked area according to the givencondition and contexts.
Revised: We conduct conditional image inpainting where a model infills a masked area according to the given condition and contexts.

Original: Figure 1 shows the example of image inpainting by RQ-Transformer (middle)and Contextual RQ-Transformer (right), when the class-condition is school bus .
Revised: Figure 1 shows the example of image inpainting by RQ-Transformer (middle) and Contextual RQ-Transformer (right), when the class-condition is school bus .

Original: RQ-Transformercannot attend to the right and bottom sides of the masked area and fails to generate a coherent imagewith given contexts.
Revised: RQ-Transformer cannot attend to the right and bottom sides of the masked area and fails to generate a coherent image with given contexts.

Original: However, our model can complete the image to be coherent with given contextsby exploiting global contexts.
Revised: However, our model can complete the image to be coherent with given contexts by exploiting global contexts.

Original: We attach more examples of image inpainting in Appendix.
Revised: We attach more examples of image inpainting in Appendix.

Original: 4.4 Ablation Study on Draft-and-Revise
Revised: 4.4 Ablation Study on Draft-and-Revise

Original: We conduct an extensive ablation study to demonstrate the effectiveness of Draft-and-Revise decodingof our framework.
Revised: We conduct an extensive ablation study to demonstrate the effectiveness of Draft-and-Revise decoding of our framework.

Original: We use Contextual RQ-Transformer with 821M parameters trained on ImageNet.
Revised: We use Contextual RQ-Transformer with 821M parameters trained on ImageNet.

Original: Quality improvement of draft images in the revise phase Figure 4(a) shows the effects of T  on draft images and their quality improvement in the revised phase with T revise = 2 and M = 2 .
Revised: Quality improvement of draft images in the revise phase Figure 4(a) shows the effects of T draft on draft images and their quality improvement in the revised phase with T revise = 2 and M = 2 .

Original: In the draft phase, FID is improved as T draft increases from 4 to 64.
Revised: In the draft phase, FID is improved as T draft increases from 4 to 64.

Original: At each inference, Contextual
Revised: At each inference, Contextual

Original: RQ-Transformer generates N/T draft code stacks in parallel, starting with the empty sequence.
Revised: RQ-Transformer generates N/T draft code stacks in parallel, starting with the empty sequence.

Original: Thus,the model with a large T draft generates a small number of code stacks at each inference and can avoidgenerating incoherent code stacks in the early stage of the draft phase.
Revised: Thus, the model with a large T draft generates a small number of code stacks at each inference and can avoid generating incoherent code stacks in the early stage of the draft phase.

Original: Although FIDs in the draftphase are worse due to low precision, we intend to increase the diversity of generated images, sincethe precision is significantly improved in the revise phase as shown in Figure 4(b).
Revised: Although FIDs in the draft phase are worse due to low precision, we intend to increase the diversity of generated images, since the precision is significantly improved in the revise phase as shown in Figure 4(b).

Original: Effect of revision iteration M and partition size T revise in the revise phase Figure 4(b) showsthe effects of the number of updates M in the revise phase on the quality and diversity of generatedimages.
Revised: Effect of revision iteration M and partition size T revise in the revise phase Figure 4(b) shows the effects of the number of updates M in the revise phase on the quality and diversity of generated images.

Original: Since the quality-diversity trade-off exists as the updates are repeated, we select M = 2 as thedefault hyperparameter to balance the precision and recall, considering that the increase of precisionstarts to slow down.
Revised: Since the quality-diversity trade-off exists as the updates are repeated, we select M = 2 as the default hyperparameter to balance the precision and recall, considering that the increase of precision starts to slow down.

Original: Interestingly, Figure 5 shows that the overall contents remain unchanged even after M > 2 .
Revised: Interestingly, Figure 5 shows that the overall contents remain unchanged even after M > 2 .

Original: Thus, we claim that Draft-and-Revise decoding does not harm the perceptual diversityof generated images throughout the revise phase despite the consistent deterioration of recall.
Revised: Thus, we claim that Draft-and-Revise decoding does not harm the perceptual diversity of generated images throughout the revise phase despite the consistent deterioration of recall.

Original: Figure 4(c) shows the effects of T revise on the quality of generated images.
Revised: Figure 4(c) shows the effects of T revise on the quality of generated images.

Original: The FIDs are significantlyimproved in the revise phase regardless of the choice of T revise , but increasing T revise slightly deteriorates FIDs.
Revised: The FIDs are significantly improved in the revise phase regardless of the choice of T revise , but increasing T revise slightly deteriorates FIDs.

Original: We remark that some code stacks of a draft can be erroneous due to its low quality,and a model with large T revise slowly updates a small number of code stacks at once in the revisephase.
Revised: We remark that some code stacks of a draft can be erroneous due to its low quality, and a model with large T revise slowly updates a small number of code stacks at once in the revise phase.

Original: Therefore, the updates with large T revise can be more influenced by the erroneous code stacks.
Revised: Therefore, the updates with large T revise can be more influenced by the erroneous code stacks.

Original: Although T revise = 2 updates half of an image at once, our draft-and-revise decoding successfullyimproves the quality of generated images, while preserving the global contexts of drafts, as shownin Figure 5.
Revised: Although T revise = 2 updates half of an image at once, our draft-and-revise decoding successfully improves the quality of generated images, while preserving the global contexts of drafts, as shown in Figure 5.

Original: The study on self-supervised learning [17] also reports similar results, where a maskedauto-encoder reconstructs the global contexts of an image after masking half of the image.
Revised: The study on self-supervised learning [17] also reports similar results, where a masked auto-encoder reconstructs the global contexts of an image after masking half of the image.

Original: [Table]
Revised: [Table]

Original: Quality-diversity control of Draft-and-Revise
Revised: Quality-diversity control of Draft-and-Revise

Original: Our
Revised: Our

Original: Draft-and-Revise decoding can effectively control thequality-diversity trade-off in generated images.
Revised: Draft-and-Revise decoding can effectively control the quality-diversity trade-off in generated images.

Original: Table 3shows FID, precision (P), and recall (R) according tothe use of classifier-free guidance [20] with a scale of1.8, while applying temperature scaling with 0.8 only tothe revise phase.
Revised: Tableshows FID, precision (P), and recall (R) according to the use of classifier-free guidance [20] with a scale of 1.8, while applying temperature scaling with 0.8 only to the revise phase.

Original: Contextual RQ-Transformer withoutthe guidance already outperforms RQ-Transformer with3.8B parameters and demonstrates the effectiveness of ourframework.
Revised: Contextual RQ-Transformer without the guidance already outperforms RQ-Transformer with 3.8B parameters and demonstrates the effectiveness of our framework.

Original: When the guidance is used for both draft andrevise phases, the precision dramatically increases but the recall decreases to 0.33.
Revised: When the guidance is used for both draft and revise phases, the precision dramatically increases but the recall decreases to 0.33.

Original: Consequently,
Revised: Consequently,

Original: FID becomes worse due to the lack of diversity in generated images.
Revised: FID becomes worse due to the lack of diversity in generated images.

Original: However, when the guidance isapplied only to the revise phase, our model achieves the lowest FID, as the quality and diversity arewell-balanced.
Revised: However, when the guidance is applied only to the revise phase, our model achieves the lowest FID, as the quality and diversity are well-balanced.

Original: Thus, the explicitly separated two phases of Draft-and-Revise can effectively controlthe issue of quality-diversity trade-off by generating diverse drafts and then improving their quality.
Revised: Thus, the explicitly separated two phases of Draft-and-Revise can effectively control the issue of quality-diversity trade-off by generating diverse drafts and then improving their quality.

Original: [Table]
Revised: [Table]

Original: Trade-off between quality and sampling speed Table 4shows that our framework can control the trade-off betweenthe quality and sampling speed according to T draft after wefix T revise = 2 and M = 2 .
Revised: Trade-off between quality and sampling speed Tableshows that our framework can control the trade-off between the quality and sampling speed according to T draft after we fix T revise = 2 and M = 2 .

Original: We generate 5,000 samples withbatch size of 100 as the previous study [23].
Revised: We generate 5,000 samples with batch size of 100 as the previous study [26].

Original: Contextual
Revised: Contextual

Original: RQ-Transformer with T draft = 8 outperforms both FID andsampling speed of VQGAN and RQ-Transformer with 1.4Bparameters.
Revised: RQ-Transformer with T draft = 8 outperforms both FID and sampling speed of VQGAN and RQ-Transformer with 1.4B parameters.

Original: Although the sampling speed becomes slowwith increased T draft , the FID scores are consistently improved.
Revised: Although the sampling speed becomes slow with increased T draft , the FID scores are consistently improved.

Original: We remark that the sampling speed with T draft = 64is about 3 × slower than RQ-Transformer, but our modeloutperforms 3.8B parameters of RQ-Transformer with rejection sampling in Table 1.
Revised: We remark that the sampling speed with T draft =is about 3 × slower than RQ-Transformer, but our model outperforms 3.8B parameters of RQ-Transformer with rejection sampling in Table 1.

Original: The results show that our framework has inexpensive costs to generatehigh-quality images, considering that rejection sampling requires generating up to 20 × more samples.
Revised: The results show that our framework has inexpensive costs to generate high-quality images, considering that rejection sampling requires generating up to 20 × more samples.

Original:  Conclusion
Revised:  Conclusion

Original: In this study, we have proposed Draft-and-Revise for an effective image generation framework with
Revised: In this study, we have proposed Draft-and-Revise for an effective image generation framework with

Original: Contextual RQ-Transformer.
Revised: Contextual RQ-Transformer.

Original: After an image is represented as a sequence of code stacks, Contextual
Revised: After an image is represented as a sequence of code stacks, Contextual

Original: RQ-Transformer is trained to infill a randomly masked sequence.
Revised: RQ-Transformer is trained to infill a randomly masked sequence.

Original: Then, Draft-and-Revise decoding isused to generate high-quality images by first generating a draft image with diverse contents and thenimproving its visual quality based on the global contexts of the draft.
Revised: Then, Draft-and-Revise decoding is used to generate high-quality images by first generating a draft image with diverse contents and then improving its visual quality based on the global contexts of the draft.

Original: Consequently, we can achievestate-of-the-art results on ImageNet and CC-3M, demonstrating the effectiveness of our framework.
Revised: Consequently, we can achieve state-of-the-art results on ImageNet and CC-3M, demonstrating the effectiveness of our framework.

Original: Our study has two main limitations to be further explored.
Revised: Our study has two main limitations to be further explored.

Original: Firstly, Draft-and-Revise decoding alwaysupdates all code stacks in the revise phase, although some code stacks might not need an update.
Revised: Firstly, Draft-and-Revise decoding always updates all code stacks in the revise phase, although some code stacks might not need an update.

Original: Infuture work, a selective method can be developed to improve the efficiency of the revise phase by asophisticated approach.
Revised: In future work, a selective method can be developed to improve the efficiency of the revise phase by a sophisticated approach.

Original: Secondly, our generative model is not validated on various downstream tasks.
Revised: Secondly, our generative model is not validated on various downstream tasks.

Original: Since masked token modeling is successful self-supervised learning for texts [8] and images [3, 17], aunified model for both generative and discriminative tasks [21] is worth exploration for future work.
Revised: Since masked token modeling is successful self-supervised learning for texts [8] and images [3, 17], a unified model for both generative and discriminative tasks [24] is worth exploration for future work.

Original:  Checklist
Revised:  Acknowledgements

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT)

Original: [Yes]
Revised: (No.2018-0-01398:

Original: (b) Did you describe the limitations of your work? [Yes]
Revised: Development of a Conversational, Self-tuning DBMS, 35%; No.2021-0-00537:

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Visual Common Sense, 30%) and the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)

Original: [No]
Revised: (No.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: NRF-2021R1A2B5B03001551, 35%) A Implementation Details A.1 Details of RQ-VAE

Original: them?
Revised: RQ-VAE [26] is a generalized version of VQ-VAE [36] and VQGAN [13], since RQ-VAE with D = 1 is equivalent to VQ-VAE or VQGAN. When D > 1 , RQ-VAE recursively discretizes the feature map of an image for a precise approximation of the feature map using the codebook. When the codebook size of RQ is K , RQ with depth D is as capable as VQ with K D size of a codebook, since RQ can represent at most K D clusters in a vector space.

Original: [Yes]
Revised: That is, if the codebook sizes are the same, RQ with D > 1 can approximate a feature vector more accurately than VQ. Thus, RQ-VAE can further reduce the spatial resolution of code map than VQ-VAE and VQ-GAN, and therefore outperforms previous autoregressive models with discrete representations. Following the previous studies [13, 26, 36], the training of RQ-VAE uses the reconstruction loss, the commitment loss, the adversarial training [22], and the LPIPS perceptual loss [23]. The codebook C of RQ-VAE is updated using the exponential moving average during training [26, 36].

Original: If you are including theoretical results...
Revised: In experiments, we use the pretrained RQ-VAE, which is publicly available 3 . The RQ-VAE uses the codebook size of 16,384 to represent a 256 × 256 resolution of an image as 8 × 8 × 4 shape of a code map. The architecture of RQ-VAE is the same as VQGAN [13] except for adding residual blocks in the encoder and the decoder to reduce the spatial resolution of the code map more than VQGAN.

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: A.2 Architecture of Contextual RQ-Transformer

Original: [N/A]
Revised: [Table]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Table 5 summarizes the architecture details of Contextual RQ-Transformers to be trained on ImageNet and CC-3M. Contextual RQTransformer consists of two compartments: bidirectional spatial transformer with N spatial selfattention blocks and depth transformer with

Original: [N/A]
Revised: N depth causal self-attention blocks.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: The dimensionality of embeddings in multi-headed selfattentions is denoted d model , while the dimensionality for each attention head is 64. A.3 Training details

Original: imental results (either in the supplemental material or as a URL)?
Revised: All Contextual RQ-Transformers are trained with AdamW optimizer with β 1 = 0 .

Original: [Yes] Our code is
Revised: 9 , β 2 = 0 .

Original: included in the supplemental material.
Revised: 95 , and weight decay 0.0001.

Original: (b) Did you specify all the training details (e
Revised: Each model is trained for 300 epochs with the cosine learning rate schedule with the initial value of 0.0001 and the final value of 0, for both ImageNet and CC-3M. We use eight NVIDIA A100 GPUs to train the models with 821M and 1.4B parameters on ImageNet and the model with 650M parameters on CC-3M, while four GPUs are used for the models with 366M parameters.

Original: g., data splits, hyperparameters,
Revised: For our model with 821M and 1.4B parameters on ImageNet, the training takes at most 10 days. A.4 Draft-and-Revise decoding details We use temperature scaling with the 0.8 scale in the revise phase and do not apply the temperature scaling in the draft phase. The classifier-free guidance is also applied only to the sampling in the revise phase.

Revised: We use 1.4, 1.8, and 2.0 scales of guidance for 371M, 821M, and 1.4B parameters of

Revised: Contextual RQ-Transformer on ImageNet, respectively.

Revised: In addition, ( M, T revise ) is (3,2), (2,2), and (2,2), respectively.

Revised: In the revise phase for Contextual RQ-Transformer on CC-3M, we use the 1.

Revised: scale of classifier-free guidance with ( M, T revise ) =

Revised: B The Compatibility of Draft-and-Revise with 16 × 16 VQGAN

Revised: Since RQ is a generalized VQ, our framework of Draft-and-Revise with Contextual RQ-Transformer is also applicable to VQ-VAE [36] and VQGAN [13].

Revised: Note that RQ-VAE with D = 1 is equivalent to

Revised: VQGAN, where their spatial resolutions of code maps are the same.

Revised: To validate that our framework

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: is also effective on the 16 × 16 shape of code map by VQGAN, we first train an RQ-VAE, which represents an image as 16 × 16 × 1 shape of code maps and has the identical architecture to VQGAN with 16 × 16 shape of code map.

Revised: Then, we train a Contextual RQ-Transformer with 350M parameters on 16 × 16 × 1 codes.

Revised: We notate the Contextual RQ-Transformer with D = 1 as Contextual VQTransformer throughout this section.

Revised: Figure 6 shows that the Draft-and-Revise decoding also generalizes to Contextual VQ-Transformer and can control the quality-diversity trade-off in the same manner as with the Contextual RQTransformer.

Revised: We fix T revise = 2 and M = 2 , and use temperature scaling of 0.8 and classifier-free guidance of 2.4 only in the revise phase.

Revised: As shown in Figure 6(a), the best performance is achieved when T draft = 256 and the corresponding FID, precision (P), and recall (R) are 6.44, 0.79, and 0.47, respectively, as reported in Table 6.

Revised: Note that Contextual VQ-Transformer outperforms the AR model such as VQGAN, although our model has 4 × fewer parameters than the AR model.

Revised: In addition, our draft-and-revise decoding with Contextual VQ-Transformer also works as well as Contextual

Revised: RQ-Transformer, showing that the iterative updates in the revise phase consistently increase precisions and decreases recalls in Figure 6(b).

Revised: Consequently, the results validate that our framework is more effective for high-quality image generation than AR modeling.

Revised: Although our framework is compatible with VQGAN, we emphasize that Contextual RQ-Transformer is more effective than Contextual VQ-Transformer.

Revised: Contextual RQ-Transformer with a similar number of parameters outperforms the Contextual VQ-Transformer in terms of FID, precision, and recall in Table 6.

Revised: In addition, Contextual RQ-Transformer has about 10 × faster speed for image generation than Contextual VQ-Transformer, since the computational complexity of self-attention is mainly determined by the sequence length.

Revised: Although the comparison of FID, precision, and recall is not entirely fair due to the different performance between RQ-VAE and VQGAN, Contextual RQTransformer can achieve state-of-the-art performance with lower computational costs than Contextual

Revised: VQ-Transformer.

Revised: Thus, masked modeling in RQ representations is more effective and efficient than in VQ representations, if equipped with our Contextual RQ-Transformer.

Revised: C Comparison with Confidence-based Mask-infilling Strategies

Revised: In this section, we examine how the selection of UPDATE in Algorithm 1 affects the performance of image generation.

Revised: First, instead of using our random updates of spatial positions, we consider

Revised: [Table]

Revised: a confidence-based mask-infilling strategy of MaskGIT [6] and denote the sampling strategy of

Revised: MaskGIT as Top-C .

Revised: At each inference step, Top-C considers the predicted confidences and determines the unmasked positions to have highly confident predictions.

Revised: Then, we also consider a mixed strategy,

Revised: Top-C-50% , which first filters out the bottom 50% confident positions, and then randomly selects the unmasked positions among the positions with top 50% high confidence.

Revised: Top-C-50% is similar to the combination of random sampling after top-k or top-p [21] filtering.

Revised: We denote our mask-infilling strategy as Random , which randomly determines the unmasked regions at each inference in the draft phase.

Revised: We use Contextual RQ-Transformer with 821M parameters and fix the parameters of

Revised: Draft-and-Revise decoding to T draft = 64 , T revise = 2 , and M = 2 and apply temperature scaling of 0.8 and classifier-free guidance of 1.8 in the revise phase.

Revised: We report FID, precision (P), and recall (R) of the generated images in the draft and revise phases.

Revised: Table 7 shows the effect of Top-C, Top-C-50%, and Random in Draft-and-Revise decoding.

Revised: Regardless of the selection of mask-infilling strategies in the draft phase, our draft-and-revise decoding consistently improves the performance of image generation after the revise phase.

Revised: The results imply that our framework of Draft-and-Revise can be effectively generalized to various approaches of mask-infilling-based image generation.

Revised: When T draft = 8 , the draft images of Top-C have better FID but worse recall than the draft images of Random.

Revised: Nonetheless, the quality of the draft images is significantly improved with the revise phase and subsequently Random outperforms Top-C in all three metrics.

Revised: Top-C-50% achieves the best FID in both the draft and revised images, but the recall of Top-C-50% is significantly lower than the recall of Random.

Revised: When T draft = 64 , the draft images of Top-C exhibit worse metrics than the draft images of Random.

Original: how they
Revised: Although Top-C-50% achieves lower FID than Top-C and Random in terms of draft images, Random outperforms Top-C and Top-C-50% after the revise phase. This supports our claim that Draft-and-Revise decoding better controls quality-diversity trade-off when drafts are generated with maximal diversity. By the visual analysis in Figure 7, we find that the limited performance of confidence-based maskinfilling strategies results from the bias of a model on high-confident predictions. That is, Top-C tends to predict only simple patterns in the early phase of mask-infilling, since the simple visual patterns are prone to have high-confident predictions.

Original: were chosen)?
Revised: This effect becomes more apparent when T draft = 64 as only the code stack with the highest confidence is included in the sample at each inference, thereby the diversity of the generated images becomes severely limited. Indeed, it is shown in Figure 7 that the confidence-based methods first infill the backgrounds with simple visual patterns, and the resulting samples exhibit low diversity of visual contents. Due to the limited diversity, the confidence-based mask-infilling strategies limit the performance of FID, although our draft-and-revise decoding further improves the visual quality of generated images after the revise phase. Thus, we conclude that our sampling strategy, which first generates diverse visual contents and then improves their quality, is effective for draft-and-revise decoding to achieve high performance of image generation.

Original: [Yes] See Sec. 4 and Appendix.
Revised: In this section, we show additional examples of generated images by our Contextual RQ-Transformer. We use 1.4B parameters of Contextual RQ-Transformer trained on ImageNet for class-conditional pine tree on a background of the sea † cherry blossom tree on a background of the sea road on the mountain during rainy season † road on the mountain during snowy season young brown bear in the wild forest † young brown bear on the seashore A cheeseburger in front of a mountain range covered with snow.

Original: (c) Did you report error bars (e
Revised: A pizza in front of a mountain range covered with snow. image generation and 654M parameters of our model trained on CC-3M for text-conditional image generation. D.1 Examples of Class-conditional Image Generation Figure 8 shows the generated images on class conditions in ImageNet.

Original: g., with respect to  the random seed after running experiments multiple times)?
Revised: Contextual RQ-Transformer generates diverse and high-quality images conditioned on given class conditions. D.2 Examples of Text-conditioned Image Generation Figure 8 shows the generated images by our model on various text conditions. We use the captions in the validation dataset † of CC-3M and change a keyword in the caption to validate the generalization power of our model on unseen compositions of texts.

Revised: Figure 8 shows that our Contextual RQTransformer can generate high-quality images on diverse compositions of text conditions, even though the text condition is unseen during training.

Revised: D.3 Examples of Image In-painting

Revised: Figure 10 shows that Draft-and-Revise decoding can also be used to inpaint the prescribed region of a given image.

Original: [No]
Revised: For image inpainting, a random partition Π used in each application of UPDATE is set to be the partition of the masked region, instead of all spatial positions. The first three rows of Figure 10 show the image inpainting results, where the original images are taken from the validation set of ImageNet, either bottom, center, or outside is masked, and the class of the original image is given as a condition. On the other hand, the last row of Figure 10 shows that our Contextual

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: RQ-Transformer is also capable of image editing via inpainting, by conditioning on a class-condition that is not the class of the original image.

Original: [Yes] See Appendix.4. If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: D.4 Examples of Mask-Infilling of Half Masked Images

Original: (a) If your work uses existing assets, did you cite the creators? [Yes] We used publiclyavailable datasets and cited them in Sec. 4. (b) Did you mention the license of  the assets? [No]
Revised: After we randomly mask the half of images in the ImageNet validation dataset, Contextual RQTransformer infills the masked regions in Figure 11.

Original: (c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Our code is included in the supplemental material.
Revised: The results show that Contextual RQ-Transformer can infill the masked regions, while preserving the global contexts of original images.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: Note that the results are also aligned to the experimental results in previous approaches [17, 6].

Original: using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: That is, our draft-and-revise decoding can preserve the global contexts of the draft images in the revise phase, although we use a small value of T revise such as 2 or 4.

Original: [N/A] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: Note that the fine-grained details can be changed after infilling masks, since our method randomly samples the codes for unmasking based on the predicted softmax distribution over codes in Eq. 9.

Original: applicable?
Revised: D.5 Societal Impact

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
Revised: Since our framework can learn large-scale datasets for a real-world applications, the model has a potential of negative social impacts by generating socially biased or violent images according to the training data.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: Thus, exhaustive filtering of training data and evaluation after training have to be conducted for avoiding social problems.