Original: Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer
Revised: Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images fromfully reflecting global contexts.
Revised: Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images fromfully reflecting global contexts.

Original: To address the issue, we propose an effective imagegeneration framework of Draft-and-Revise with Contextual RQ-transformer toconsider global contexts during the generation process.
Revised: To address the issue, we propose an effective imagegeneration framework of Draft-and-Revise with Contextual RQ-transformer toconsider global contexts during the generation process.

Original: As a generalized VQ-VAE,
Revised: As a generalized VQ-VAE,

Original: RQ-VAE first represents a high-resolution image as a sequence of discrete codestacks.
Revised: RQ-VAE first represents a high-resolution image as a sequence of discrete codestacks.

Original: After code stacks in the sequence are randomly masked, Contextual RQTransformer is trained to infill the masked code stacks based on the unmaskedcontexts of the image.
Revised: After code stacks in the sequence are randomly masked, Contextual RQTransformer is trained to infill the masked code stacks based on the unmaskedcontexts of the image.

Original: Then,  Contextual RQ-Transformer uses our two-phasedecoding, Draft-and-Revise , and generates an image, while  exploiting the globalcontexts of the image during the generation process.
Revised: Then, we propose the two-phase decoding, Draft-and-Revise,for Contextual RQ-Transformer to generates an image, while fully exploiting theglobal contexts of the image during the generation process.

Original: Specifically.
Revised: Specifically.

Original: in the draftphase, our model first focuses on generating diverse images despite rather lowquality.
Revised: in thedraft phase, our model first focuses on generating diverse images despite ratherlow quality.

Original: Then, in the revise phase, the model iteratively improves the quality ofimages, while preserving the global contexts of generated images.
Revised: Then, in the revise phase, the model iteratively improves the quality ofimages, while preserving the global contexts of generated images.

Original: In experiments,our method achieves state-of-the-art results on conditional image generation.
Revised: In experiments,our method achieves state-of-the-art results on conditional image generation.

Original: Wealso validate that the Draft-and-Revise decoding can achieve high performance byeffectively controlling the quality-diversity trade-off in image generation.
Revised: Wealso validate that the Draft-and-Revise decoding can achieve high performance byeffectively controlling the quality-diversity trade-off in image generation.

Original:  Introduction
Revised:  Introduction

Original: Learning discrete representations of images enables autoregressive (AR) models to achieve promisingresults on high-resolution image generation.
Revised: Learning discrete representations of images enables autoregressive (AR) models to achieve promisingresults on high-resolution image generation.

Original: Here, an image is encoded into a feature map, which isrepresented as a sequence of discrete codes [13, 34] or code stacks [23].
Revised: Here, an image is encoded into a feature map, which isrepresented as a sequence of discrete codes [13, 33] or code stacks [23].

Original: Then, an AR model generatesa sequence of codes in the raster scan order and decodes the codes into an image.
Revised: Then, an AR model generatesa sequence of codes in the raster scan order and decodes the codes into an image.

Original: Consequently, ARmodels show high performance and scalability on large-scale datasets [13, 23, 27].
Revised: Consequently, ARmodels show high performance and scalability on large-scale datasets [13, 23, 26].

Original: Despite the promising results of AR models, we postulate that the ability of AR models is limited due tothe lack of considering global contexts in the generation process.
Revised: Despite the promising results of AR models, we postulate that the ability of AR models is limited due tothe lack of considering global contexts in the generation process.

Original: Specifically, since AR models generateimages by sequentially predicting the next code andattending to only precedent codes generated, they neither exploit the later part of the generated image norconsider the global contexts during generation.
Revised: Specifically, since AR models generateimages by sequentially predicting the next code andattending to only precedent codes generated, they neither exploit the later part of the generated image norconsider the global contexts during generation.

Original: Forexample, Figure 1 (middle) shows that an AR modelfails to generate a coherent image, when it is askedto inpaint the masked region of Figure 1 (left) with a school bus.
Revised: Forexample, Figure 1 (middle) shows that an AR modelfails to generate a coherent image, when it is askedto inpaint the masked region of Figure 1 (left) with a school bus.

Original: Such a failure is due to the inabilityof AR models to refer to the context of traffic lane on the right side of the masked region.
Revised: Such a failure is due to the inabilityof AR models to refer to the context of traffic lane on the right side of the masked region.

Original: To address this issue, we propose an effective image generation framework, Draft-and-Revise , with acontextual transformer to exploit the global contexts of images.
Revised: To address this issue, we propose an effective image generation framework, Draft-and-Revise , with acontextual transformer to exploit the global contexts of images.

Original: Given a randomly masked image, thecontextual transformer is first trained to infill the masks by bidirectional self-attentions similarly to
Revised: Given a randomly masked image, thecontextual transformer is first trained to infill the masks by bidirectional self-attentions similarly to

Original: BERT [8].
Revised: BERT [8].

Original: To fully leverage the contextual prediction in generation, we propose Draft-and-Revisedecoding which has two phases, draft and revise , imitating the image generation process of a humanexpert who draws a draft first and iteratively revises the draft to improve its quality.
Revised: To fully leverage the contextual prediction in generation, we propose Draft-and-Revisedecoding which has two phases, draft and revise , imitating the image generation process of a humanexpert who draws a draft first and iteratively revises the draft to improve its quality.

Original: In the draft phase,the model first infills an empty image to generate a draft image with diverse contents despite therather low-quality.
Revised: In the draft phase,the model first infills an empty image to generate a draft image with diverse contents despite therather low-quality.

Original: In the revise phase, the visual quality of the draft is iteratively improved, while theglobal contexts of the draft are preserved and exploited.
Revised: In the revise phase, the visual quality of the draft is iteratively improved, while theglobal contexts of the draft are preserved and exploited.

Original: Consequently, our Draft-and-Revise withcontextual transformer effectively generates high-quality images with diverse contents.
Revised: Consequently, our Draft-and-Revise withcontextual transformer effectively generates high-quality images with diverse contents.

Original: We use residual-quantized VAE (RQ-VAE) [23] to implement our image generation framework, since
Revised: We use residual-quantized VAE (RQ-VAE) [23] to implement our image generation framework, since

Original: RQ-VAE generalizes vector-quantized VAE (VQ-VAE)
Revised: RQ-VAE generalizes vector-quantized VAE (VQ-VAE)

Original: [34] by representing an image as a sequenceof code stacks instead of a sequence of codes.
Revised: [33] by representing an image as a sequenceof code stacks instead of a sequence of codes.

Original: Then, we propose Contextual RQ-Transformer as acontextual transformer for masked code stack modeling of RQ-VAE.
Revised: Then, we propose Contextual RQ-Transformer as acontextual transformer for masked code stack modeling of RQ-VAE.

Original: Specifically, given a sequenceof randomly masked code stacks, Contextual RQ-Transformer first uses a bidirectional transformer tocapture the global contexts of unmasked code stacks.
Revised: Specifically, given a sequenceof randomly masked code stacks, Contextual RQ-Transformer first uses a bidirectional transformer tocapture the global contexts of unmasked code stacks.

Original: Based on the global contexts, the masked codestacks are predicted in parallel, while the codes in each masked code stack are sequentially predicted.
Revised: Based on the global contexts, the masked codestacks are predicted in parallel, while the codes in each masked code stack are sequentially predicted.

Original: In experiments, our Draft-and-Revise framework with Contextual RQ-Transformer achieves state-ofthe-art results on conditional image generation and remarkable improvements on image inpainting.
Revised: In experiments, our Draft-and-Revise framework with Contextual RQ-Transformer achieves state-ofthe-art results on conditional image generation and remarkable improvements on image inpainting.

Original: Inaddition, we demonstrate that Draft-and-Revise decoding can effectively control the quality-diversitytrade-off in image generation to achieve high performance.
Revised: Inaddition, we demonstrate that Draft-and-Revise decoding can effectively control the quality-diversitytrade-off in image generation to achieve high performance.

Original: The main contributions of this paper are summarized as follows.
Revised: The main contributions of this paper are summarized as follows.

Original: We propose an intuitive andpowerful framework,Draft-and-Revise , for image generation based on a bidirectional transformer. 2) We propose Contextual RQ-Transformer  for masked code stack modeling of RQ-VAE and empiricallyshow that the proposed model with Draft-and-Revise decoding achieves state-of-the-art results onclass- and text-conditional image generation benchmarks.
Revised: We propose an intuitive andpowerful framework, Contextual RQ-Transformer , for masked code stack modeling of RQ-VAEbased on a bidirectional transformer. We propose a novel two-phase decoding Draft-and-Revise ,for bidirectional transformers to fully exploit the global contexts during image generation andachieve state-of-the-art results on class- and text-conditional image generation benchmarks.

Original: An extensive ablation study validatesthe effectiveness of Draft-and-Revise decoding on controlling the quality-diversity trade-off and itscapability to generate high-quality images with diverse contents.
Revised: 3) Anextensive ablation study validates the effectiveness of Draft-and-Revise decoding on controlling thequality-diversity trade-off and its capability to generate high-quality images with diverse contents.

Original:  Related Work
Revised:  Related Work

Original: Discrete Representation for Image Generation By representing an image as a sequence of codes,
Revised: Discrete Representation for Image Generation By representing an image as a sequence of codes,

Original: VQ-VAE [34] becomes an important part for high-resolution image generation [6, 10, 15, 23, 27, 34],but suffers from low quality of reconstructed images.
Revised: VQ-VAE [33] becomes an important part for high-resolution image generation [6, 10, 15, 23, 26, 33],but suffers from low quality of reconstructed images.

Original: However, VQGAN [13] significantly improvesthe perceptual quality of reconstructed images by adding the adversarial and perceptual losses into thetraining objective of VQ-VAE.
Revised: However, VQGAN [13] significantly improvesthe perceptual quality of reconstructed images by adding the adversarial and perceptual losses into thetraining objective of VQ-VAE.

Original: As a generalized approach of VQ-VAE and VQGAN, RQ-VAE [23]represents an image as a sequence of code stacks, which consists of ordered codes, and reduces thesequence length, while preserving the reconstruction quality.
Revised: As a generalized approach of VQ-VAE and VQGAN, RQ-VAE [23]represents an image as a sequence of code stacks, which consists of ordered codes, and reduces thesequence length, while preserving the reconstruction quality.

Original: Then, RQ-Transformer [23] achieveshigh performance with lower computational costs on generating high-resolution images.
Revised: Then, RQ-Transformer [23] achieveshigh performance with lower computational costs on generating high-resolution images.

Original: However, asan AR model of RQ-VAE, RQ-Transformer cannot capture the global contexts of generated images.
Revised: However, asan AR model of RQ-VAE, RQ-Transformer cannot capture the global contexts of generated images.

Original:  Tasks with Bidirectional Transformers To overcome the limitation of AR models onunidirectional architecture, bidirectional transformers have been used for generative tasks.
Revised: Generation Tasks with Bidirectional Transformers To overcome the limitation of AR modelson unidirectional architecture, bidirectional transformers have been studied for generative tasks.

Original: Similarto the pretraining objective of BERT [8], a bidirectional transformer is trained to infill a randommask.
Revised: After a bidirectional transformer is trained to infill a random mask as the masked token modeling of

Original: Then, accompanied with an iterative decoding method [14, 25, 31, 35], the model can generatetexts [14], images [6], or videos [16, 37].
Revised: BERT [8], an effective decoding method has been proposed for bidirectional transformers to generatetexts [14, 30], images [6, 35], or videos [16].

Original: Recently, discrete diffusion models [1, 4, 12, 15] alsouses bidirectional transformers to generate an image. Given a partially corrupted by random codereplacement [1, 12] or randomly masked [1, 4, 15] sequence of codes, diffusion models are trained togradually denoise thecorrupted codes or infill the masks.
Revised: For image generation, M6-UFC [35] first incorporatesthe masked token modeling in the codes of VQGAN [13] for image generation. MaskGIT [6]proposes the confidence-based mask generation to achieve high performance on ImageNet [7].

Original: The training of discrete diffusion modelswith an absorbing state [1] is the same to infill randomly masked sequence [6, 4].
Revised: Similar to M6-UFC [35] and MaskGIT [6], we also incorporate the masked token modeling of

Original:  However, differentfrom the reverse process of diffusion models, our decodingmethod has explicit two phases to generatehigh-quality images with diverse contents.  Draft-and-Revise Framework for Effective Image Generation
Revised: RQ-VAE in a bidirectional transformer for image generation. However, instead of the confidencebased decoding [6, 35], we propose a novel two-phase decoding for a bidirectional transformerto achieve state-of-the-art performance and validate its effectiveness.

Original: In this section, we propose our Draft-and-Revise framework for effective image generation usingbidirectional contexts of images. We first review RQ-VAE [23] as a generalization of VQ-VAE. Then, [Figure]
Revised: Recently, discrete diffusionmodels [1, 4, 12, 15] also uses bidirectional transformers to generate an image, and the trainingwith an absorbing state [1] is also the same to infill randomly masked sequence [4, 6].

Original: Tokenizing
Revised: However,different from the reverse process of diffusion models, our decoding method has explicit two phasesto generate high-quality images with diverse contents.

Original: [Figure]
Revised: [Figure]

Original: RQ-VAE
Revised: RQ-VAE

Original: [Figure]
Revised: [Figure]

Original: Random
Revised: In this section, we propose our Draft-and-Revise framework for effective image generation usingbidirectional contexts of images.

Original: [Figure] RQ [Figure] Transformer
Revised: We first review RQ-VAE [23] as a generalization of VQ-VAE.

Original: [Figure] we propose Contextual RQ-Transformer which is trained to infill a randomly masked sequence ofcode stacks of RQ-VAE by understanding bidirectional contexts of unmasked parts in the sequence.
Revised: Then,we propose Contextual RQ-Transformer which is trained to infill a randomly masked sequence ofcode stacks of RQ-VAE by understanding bidirectional contexts of unmasked parts in the sequence.

Original: Lastly, we propose draft-and-revise decoding for a bidirectional transformer to effectively generatehigh-quality images exploiting global contexts of images.
Revised: Lastly, we propose draft-and-revise decoding for a bidirectional transformer to effectively generatehigh-quality images exploiting global contexts of images.

Original: Figure 2 provides the overview of ourproposed framework, including Contextual RQ-Transformer and Draft-and-Revise decoding.
Revised: Figure 2 provides the overview of ourproposed framework, including Contextual RQ-Transformer and Draft-and-Revise decoding.

Original: 3.1 Residual-Quantized Variational Autoencoder (RQ-VAE)
Revised: 3.1 Residual-Quantized Variational Autoencoder (RQ-VAE)

Original: RQ-VAE [23] represents an image as a sequence of code stacks.
Revised: RQ-VAE [23] represents an image as a sequence of code stacks.

Original: Let a codebook C
Revised: Let a codebook C

Original: = { ( k, e ( k )) }
Revised: = { ( k, e ( k )) }

Original: k ∈
Revised: k ∈

Original: [ K ]include pairs of a code k and its code embedding e ( k ) ∈ R n z , where K = |C| is the codebook sizeand n z is the dimensionality of e ( k ) .
Revised: [ K ]include pairs of a code k and its code embedding e ( k ) ∈ R n z , where K = |C| is the codebook sizeand n z is the dimensionality of e ( k ) .

Original: Given a vector z ∈ R n z , Q ( z ; C ) is defined as the code of z :
Revised: Given a vector z ∈ R n z , Q ( z ; C ) is defined as the code of z :

Original: [Equation]
Revised: [Equation]

Original: Then, RQ with depth D represents a vector as a code stack which consists of D codes:
Revised: Then, RQ with depth D represents a vector as a code stack which consists of D codes:

Original: [Equation]
Revised: [Equation]

Original: where k d is the d -th code of z .
Revised: where k d is the d -th code of z .

Original: Specifically, RQ first initializes the 0 -th residual vector as r 

Revised: Specifically, RQ first initializes the 0 -th residual vector as r

Original: =
Revised: =

Original: z , andthen recursively discretizes a residual vector r d − 1 and computes the next residual vector r d as 
Revised: z , andthen recursively discretizes a residual vector r d − 1 and computes the next residual vector r d as [Equation] d [Equation]

Revised: Q

Original: [Equation]
Revised: [Equation]

Original: for d ∈ [ D ] .
Revised: for d ∈ [ D ] .

Original: Finally, z is approximated by the sum of the D code embeddings ˆ z := (cid:80) Dd =1 e ( k d ) .
Revised: Finally, z is approximated by the sum of the D code embeddings ˆ z := (cid:80) Dd =1 e ( k d ) .

Original: Weremark that RQ is a generalized version of VQ, as RQ with D =1 is equivalent to VQ.
Revised: Weremark that RQ is a generalized version of VQ, as RQ with D =1 is equivalent to VQ.

Original: For D > 1 ,
Revised: For D > 1 ,

Original: RQ conducts a finer approximation of z as the quantization errors are sequentially reduced as dincreases.
Revised: RQ conducts a finer approximation of z as the quantization errors are sequentially reduced as dincreases.

Original: Here, the coarse-to-fine approximation ensures the D codes to be sequentially dependent.
Revised: Here, the coarse-to-fine approximation ensures the D codes to be sequentially dependent.

Original: RQ-VAE represents an image as a map of code stacks.
Revised: RQ-VAE represents an image as a map of code stacks.

Original: Specifically, a given image X is first convertedto a low-resolution feature map Z
Revised: Specifically, a given image X is first convertedto a low-resolution feature map Z

Original: = E ( X ) ∈
Revised: = E ( X ) ∈

Original: R H
Revised: R H

Original: × W × n z , and then each feature vector Z hw at spatialposition ( h, w ) is discretized into a code stack by RQ with depth D .
Revised: × W × n z , and then each feature vector Z hw at spatialposition ( h, w ) is discretized into a code stack by RQ with depth D .

Original: As a result, we get a map ofcode stacks S ∈
Revised: As a result, we get a map ofcode stacks S ∈

Original: [ K ] H × W × D .
Revised: [ K ] H × W × D .

Original: Further details of RQ-VAE are referred to Appendix.
Revised: Further details of RQ-VAE are referred to Appendix.

Original: 3.2 Contextual Transformer for Image Generation with Global Contexts
Revised: 3.2 Contextual Transformer for Image Generation with Global Contexts

Original: As a bidirectional transformer for RQ-VAE, we propose Contextual RQ-Transformer for imagegeneration based on a contextual understanding of images.
Revised: As a bidirectional transformer for RQ-VAE, we propose Contextual RQ-Transformer for imagegeneration based on a contextual understanding of images.

Original: First, we adopt the pretraining of BERT [8]to formulate a masked code stack modeling of RQ-VAE.
Revised: First, we adopt the pretraining of BERT [8]to formulate a masked code stack modeling of RQ-VAE.

Original: Then, we introduce how Contextual RQTransformer infills the randomly masked code stacks after reading the given contextual information.
Revised: Then, we introduce how Contextual RQTransformer infills the randomly masked code stacks after reading the given contextual information.

Original: 3.2.1 Masked Code Stack Modeling of RQ-VAE
Revised: 3.2.1 Masked Code Stack Modeling of RQ-VAE

Original: By adopting the pretraining of BERT [8], we formulate the masked code stack modeling of RQ-VAEwith a contextual transformer to generate an image by iterative mask-infilling as non-AR models [14].
Revised: By adopting the pretraining of BERT [8], we formulate the masked code stack modeling of RQ-VAEwith a contextual transformer to generate an image by iterative mask-infilling as non-AR models [14].

Original: We first convert the map S ∈
Revised: We first convert the map S ∈

Original: [ K ] H × W × D into a sequence of code stacks S ′ ∈
Revised: [ K ] H × W × D into a sequence of code stacks S ′ ∈

Original: [ K ] N × D using theraster-scan ordering, where N = HW and S ′ n = ( S ′ n 1 , · · · , S ′ nD ) ∈
Revised: [ K ] N × D using theraster-scan ordering, where N = HW and S ′ n = ( S ′ n 1 , · · · , S ′ nD ) ∈

Original: [ K ] D for n ∈ [ N ] .
Revised: [ K ] D for n ∈ [ N ] .

Original: We denote
Revised: We denote

Original: S ′ as S for the brevity of notation.
Revised: S ′ as S for the brevity of notation.

Original: A mask vector m is defined as a binary vector m ∈ { 0 , 1 } N toindicate the spatial positions to be masked.
Revised: A mask vector m is defined as a binary vector m ∈ { 0 , 1 } N toindicate the spatial positions to be masked.

Original: Then, the masked sequence S \ m of S by m is defined as
Revised: Then, the masked sequence S \ m of S by m is defined as

Original: [Equation]
Revised: [Equation]

Original: where [ MASK ] is a mask token to substitute for S nd if m n = 1 .
Revised: where [ MASK ] is a mask token to substitute for S nd if m n = 1 .

Original: Given a random mask vectorm ∼ q ( m ) , the masked code stacks given S \ m are modeled as
Revised: Given a random mask vectorm ∼ q ( m ) , the masked code stacks given S \ m are modeled as

Original: [Equation]
Revised: [Equation]

Original: where q ( m ) is a mask distribution where the masking portion (cid:80) Nn
Revised: where q ( m ) is a mask distribution where the masking portion (cid:80) Nn

Original: =

Revised: =

Original: m i /N in (0 , 1] as well as themasking positions are randomly chosen.
Revised: m i /N in (0 , 1] as well as themasking positions are randomly chosen.

Original: Instead of fixing the portion to 15% as in BERT, training amodel with a random masking portion from (0 , 1] enables the model to generate new images based onvarious masking patterns including m n = 1 for all n .
Revised: Instead of fixing the portion to 15% as in BERT, training amodel with a random masking portion from (0 , 1] enables the model to generate new images based onvarious masking patterns including m n = 1 for all n .

Original: We explain the details of q ( m ) in Section 3.2.3.
Revised: We explain the details of q ( m ) in Section 3.2.3.

Original: The left-hand side of Eq.
Revised: The left-hand side of Eq.

Original: 5 implies that all masked code stacks can be decoded in parallel, afterextracting contextual information from S \ m .
Revised: 5 implies that all masked code stacks can be decoded in parallel based onthe contexts of S \ m .

Original: If D = 1 , Eq. 5 becomes equivalent to conventionalmasked token modeling of texts [8] and images [6, 16] where a single token at each masked positionis predicted.
Revised: If D = 1 , Eq. 5 becomes equivalent to conventional masked token modeling oftexts [8] and images [6, 16] where a single token at each masked position is predicted.

Original: For D > 1 , the D codes of S n are autoregressively predicted, as they are sequentiallycomputed in Eq.
Revised: For D > 1 , the D codes of S n are autoregressively predicted, as they are sequentially computed in Eq.

Original: 3 for a coarse-to-fine approximation and hence well-suited for an AR prediction.
Revised: 3 for a coarseto-fine approximation and hence well-suited for an AR prediction.

Revised: We show that the effectiveness ofour framework is generalizable regardless of D in Appendix B.

Revised: In addition, the parallel decoding cancontrol the trade-off between the quality and sampling speed of image generation.

Original: 3.2.2 Contextual RQ-Transformer
Revised: 3.2.2 Contextual RQ-Transformer

Original: We modify the previous RQ-Transformer [23] for masked code stack modeling with bidirectionalcontexts in Eq.
Revised: We modify the previous RQ-Transformer [23] for masked code stack modeling with bidirectionalcontexts in Eq.

Original: Contextual RQ-Transformer consists of Bidirectional Spatial Transformer and
Revised: Contextual RQ-Transformer consists of Bidirectional Spatial Transformer and

Original: Depth Transformer : Bidirectional Spatial Transformer understands contextual information in theunmasked code stacks using bidirectional self-attentions, and Depth Transformer infills the maskedcode stacks in parallel, by autoregressively predicting the D codes at each position.
Revised: Depth Transformer : Bidirectional Spatial Transformer understands contextual information in theunmasked code stacks using bidirectional self-attentions, and Depth Transformer infills the maskedcode stacks in parallel, by autoregressively predicting the D codes at each position.

Original: Bidirectional Spatial Transformer Given a masked sequence of code stacks S \ m , bidirectionalspatial transformer first embeds  S \ m using the code embeddings of RQ-VAE as
Revised: Bidirectional Spatial Transformer Given a masked sequence of code stacks S \ m , bidirectionalspatial transformer first embeds the masked sequence S \ m using the code embeddings of RQ-VAE as

Original: [Equation]
Revised: [Equation]

Original: where PE N ( n ) is an embedding for position n , and e [ MASK ] ∈ R n z is an embedding for [ MASK ] .
Revised: where PE N ( n ) is an embedding for position n , and e [ MASK ] ∈ R n z is a mask embedding.

Original: Then, the bidirectional self-attention blocks, f spatial θ , extracts the context vector h n to predict  S n as
Revised: Then, thebidirectional self-attention blocks, f spatial θ , extracts the context vector h n to predict a code stack S n ,

Original: [Equation]
Revised: [Equation]

Original:  Depth Transformer Depth transformer autoregressively predicts S n = ( S n 1 , · · · , S nD ) at amasked position.
Revised: [Equation] [Equation] Depth Transformer The code stack at a masked position S n = ( S n 1 , · · · , S nD ) is autoregressivelypredicted.

Original: The input of depth transformer ( v nd )
Revised: The input of depth transformer ( v nd )

Original: Dd =1 is defined as
Revised: Dd =1 is defined as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where PE D ( d ) is the positional embedding for depth d .
Revised: where PE D ( d ) is the positional embedding for depth d .

Original: Then, depth transformer f depth θ , whichconsists of causal attention blocks, outputs the logit p nd to predict  S nd  asp nd = f depth θ ( v n 1 , · · · , v nd ) and p θ ( S nd = k | S n,<d , S \ m )
Revised: Then, depth transformer f depth θ , whichconsists of causal attention blocks, outputs the logit p nd to predict the d -th code S nd at position n asp nd = [Equation] depth θ [Equation]

Original: =
Revised: [Equation]

Original: softmax( p nd ) k .
Revised: v nd ) and p θ [Equation] S nd

Original: [Equation]
Revised: [Equation]

Original: We remark that the architecture of Contextual RQ-Transformer subsumes bidirectional transformers depth
Revised: We remark that the architecture of Contextual RQ-Transformer subsumes bidirectional transformers.

Revised: Specifically, RQ-Transformer with the depth D = 1 is equivalent to a bidirectional transformer sincethe depth transformer becomes a multilayer perceptron with layer normalization [2].

Original: Require: A sequence of code stacks S , a partition Π = ( m 1 , · · · , m T ) , a model θ 1: for t = 1 , · · · , T do 2:
Revised: Require: A sequence of code stacks S , a partition Π = ( m 1 , · · · , m T ) , a model θ 1: for t = 1 , · · · , T do 2:

Original: Sample S n ∼ p θ ( S n | S \ m t ) ∀ n : m tn =
Revised: Sample S n ∼ p θ ( S n | S \ m t ) ∀ n : m tn =

Original: ▷ update the codes at masked positions 3: end for 4: return S
Revised: ▷ update the codes at masked positions 3: end for 4: return S

Original: Require: Partition sampling distributions p draft and p rev , the number of revision iterations M /* draft phase */ 1: S empty ← ([ MASK ] , · · · , [ MASK ]) N ▷ initialize empty code map 2: Sample Π ∼ p ( Π ; T draft ) 3: S draft ← UPDATE ( S empty , Π ; θ ) ▷ generate a draft code map /* revision phase */ 4: S 0 ← S draft 5: for m = 1 , · · · , M do 6:
Revised: Require: Partition sampling distributions p draft and p rev , the number of revision iterations M /* draft phase */ 1: S empty ← ([ MASK ] , · · · , [ MASK ]) N ▷ initialize empty code map 2: Sample Π ∼ p ( Π ; T draft ) 3: S draft ← UPDATE ( S empty , Π ; θ ) ▷ generate a draft code map /* revision phase */ 4: S 0 ← S draft 5: for m = 1 , · · · , M do 6:

Original: Sample Π ∼ p ( Π ; T revise ) 7:
Revised: Sample Π ∼ p ( Π ; T revise ) 7:

Original: S m ← UPDATE ( S m − 1 , Π ; θ ) ▷ iteratively revise the code map 8: end for 9: return S M
Revised: S m ← UPDATE ( S m − 1 , Π ; θ ) ▷ iteratively revise the code map 8: end for 9: return S M

Original: 3.2.3 Training of Contextual RQ-Transformer
Revised: 3.2.3 Training of Contextual RQ-Transformer

Original: For the training of Contextual RQ-Transformer, let us define a mask distribution q ( m ) with a maskscheduling function γ .
Revised: For the training of Contextual RQ-Transformer, let us define a mask distribution q ( m ) with a maskscheduling function γ .

Original: Following previous approaches [6, 14, 16],  γ is chosen to be decreasing andto satisfy γ (0) = 1 and γ (1) = 0 .
Revised: Following previous approaches [6, 14, 16], the scheduling function γ is chosento be decreasing and to satisfy γ (0) = 1 and γ (1) = 0 .

Original: Then,  m ∼ q ( m ) is specified asr ∼ Unif([0 , 1)) and m ∼ Unif( { m : | m | = ⌈ γ ( r ) · N ⌉} ) , (10) where | m | = (cid:80) n ∈ [ N ] m n is the count of masked positions.
Revised: Then, a mask m ∼ q ( m ) is specified asr ∼ Unif([0 , 1)) and m ∼ Unif( { m : | m | = ⌈ γ ( r ) · N ⌉} ) , (10) where | m | = (cid:80) n ∈ [ N ] m n is the count of masked positions.

Original: Finally, the training objective of
Revised: Finally, the training objective of

Original: Contextual RQ-Transformer is to minimize the negative log-likelihood of masked code stacks:[Equation] m [Equation]
Revised: Contextual RQ-Transformer is to minimize the negative log-likelihood of masked code stacks:

Original: (cid:34)

Original: [Equation]

Original: n

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]

Original: (cid:88)

Original: d

Original: [Equation]
Revised: [Equation]

Original: 3.3 Draft-and-Revise: Two-Phase Decoding with Global Contexts of Generated Imaegs
Revised: 3.3 Draft-and-Revise: Two-Phase Decoding with Global Contexts of Generated Imaegs

Original: We propose a  decoding algorithm, Draft-and-Revise , which uses Contextual RQ-Transformer toeffectively generate high-quality images with diverse visual contents.
Revised: We propose a novel decoding algorithm, Draft-and-Revise , for Contextual RQ-Transformer toeffectively generate high-quality images with diverse visual contents.

Revised: We remark that our effectivedecoding method is required for bidirectional transformers to fully exploit the global contexts ofimages and achieve high performance of image generation.

Original: We introduce the details of 
Revised: Here, we introduce the details of our

Original: Draft-and-Revise decoding and then explain how the two-phase decoding can effectively control thequality-diversity trade-off of generated images.
Revised: Draft-and-Revise decoding and then explain how the two-phase decoding of Draft-and-Revise caneffectively control the quality-diversity trade-off of generated images.

Original: We define a partition Π
Revised: We define a partition Π

Original: =
Revised: =

Original: ( m 1 , · · · , m T ) as a collection of pairwise disjoint T mask vectors tocover all spatial positions, where (cid:80) Tt =1 m tn = 1 for all n ∈ [ N ] .
Revised: ( m 1 , · · · , m T ) as a collection of pairwise disjoint T mask vectors tocover all spatial positions, where (cid:80) Tt =1 m tn = 1 for all n ∈ [ N ] .

Original: A partition Π is sampled from thedistribution p ( Π ; T ) , which is the uniform distribution over all balanced partitions with size T :
Revised: A partition Π is sampled from thedistribution p ( Π ; T ) , which is the uniform distribution over all balanced partitions with size T :

Original: [Equation]
Revised: [Equation]

Original: We first define a procedure UPDATE ( S , Π ) to update  S as described in Algorithm 1, which updates S n with m tn = 1 for t ∈ [ T ] .
Revised: We first define a procedure UPDATE ( S , Π ) to update the sequence S as described in Algorithm 1,which updates S n with m tn = 1 for t ∈ [ T ] .

Original: Then, Draft-and-Revise decoding in Algorithm 2 generates a draft fromthe empty sequence of code stacks and improves the quality of the draft.
Revised: Then, Draft-and-Revise decoding in Algorithm 2generates a draft from the empty sequence of code stacks and improves the quality of the draft.

Original: Draft phase In the draft phase, our model gradually infills the empty sequence of code stacks togenerate a draft image, considering the global contexts of infilled code stacks.
Revised: Draft phase In the draft phase, our model gradually infills the empty sequence of code stacks togenerate a draft image, considering the global contexts of infilled code stacks.

Original: Let S empty be an emptysequence of code stacks with S empty n =
Revised: Let S empty be an emptysequence of code stacks with S empty n =

Original: [ MASK ] D for all n .
Revised: [ MASK ] D for all n .

Original: Given a partition size T draft , our modelgenerates a draft image as
Revised: Given a partition size T draft , our modelgenerates a draft image as

Original: [Equation]

Original:  Π [Equation] θ ) where Π ∼ p ( Π ; T draft ) .
Revised: S draft = UPDATE ( S empty , Π ; θ ) where Π ∼ p ( Π ; T draft ) .

Original: [Table]
Revised: [Table]

Original: Revise phase The generated draft S draft is repeatedly revised to improve the visual quality of theimage, while preserving the overall structure of the draft.
Revised: Revise phase The generated draft S draft is repeatedly revised to improve the visual quality of theimage, while preserving the overall structure of the draft.

Original: Given a partition size T revise and the numberof updates M , the draft S 

Revised: Given a partition size T revise and the numberof updates M , the draft S

Original: =
Revised: =

Original: S draft is repeatedly updated M times as
Revised: S draft is repeatedly updated M times as

Original: S m = UPDATE ( S m − 1 , Π ; θ ) where Π ∼ p
Revised: S m = UPDATE ( S m − 1 , Π ; θ ) where Π ∼ p

Original: ( Π ; T revise ) for m = 1 , · · · , M .
Revised: ( Π ; T revise ) for m = 1 , · · · , M .

Original: Note that Draft-and-Revise is not a tailored method, since we can adopt any mask-infilling-basedgeneration method [4, 6] for UPDATE in Algorithm 1.
Revised: Note that Draft-and-Revise is not a tailored method, since we can adopt any mask-infilling-basedgeneration method [4, 6] for UPDATE in Algorithm 1.

Original: For example, confidence-based decoding [6,16], which iteratively updates S from high-confidence to low-confidence predictions, can be used for
Revised: For example, confidence-based decoding [6,16], which iteratively updates S from high-confidence to low-confidence predictions, can be used for

Original: UPDATE.
Revised: UPDATE.

Original: However, we find that confidence-based decoding generates low-diversity images withoversimplified contents, since a model tends to predict simple visual patterns with high confidence.
Revised: However, we find that confidence-based decoding generates low-diversity images withoversimplified contents, since a model tends to predict simple visual patterns with high confidence.

Original: In addition, confidence-based decoding often leads to biased unmasking patterns, which are not usedin training, as shown in Appendix.
Revised: Inaddition, confidence-based decoding often leads to biased unmasking patterns, which are not used intraining. We attach the detailed discussion about confidence-based decoding in Appendix C.

Original: Thus, we use a uniformly random partition Π in UPDATE as themost simplified rule, leaving investigations on sophisticated update methods as future work.
Revised: Thus, weuse a uniformly random partition Π in UPDATE as the most simplified rule, leaving investigationson sophisticated update methods as future work.

Original: We postulate that our Draft-and-Revise can generate high-quality images with diverse contents byexplicitly dividing two phases.
Revised: We postulate that our Draft-and-Revise can generate high-quality images with diverse contents byexplicitly dividing two phases.

Original: Specifically, a model first generates draft images with diverse visualcontents despite the rather low quality of drafts.
Revised: Specifically, a model first generates draft images with diverse visualcontents despite the rather low quality of drafts.

Original: After semantically diverse images are generated asdrafts, we use sampling strategies such as temperature scaling [19] and classifier-free guidance [20]in the revise phase to improve the visual quality of the drafts, while preserving the major semanticcontents in drafts.
Revised: After semantically diverse images are generated asdrafts, we use sampling strategies such as temperature scaling [19] and classifier-free guidance [20]in the revise phase to improve the visual quality of the drafts, while preserving the major semanticcontents in drafts.

Original: Thus, our method can improve the performance of image generation by effectivelycontrolling the quality-diversity trade-off.
Revised: Thus, our method can improve the performance of image generation by effectivelycontrolling the quality-diversity trade-off.

Original: In addition, we emphasize that the two-phased decoding isintuitive and resembles the image generation process of human experts, who repeatedly refine theirworks to improve the quality after determining the overall contents first.
Revised: In addition, we emphasize that the two-phased decoding isintuitive and resembles the image generation process of human experts, who repeatedly refine theirworks to improve the quality after determining the overall contents first.

Original:  Experiments
Revised:  Experiments

Original: In this section, we show that our Draft-and-Revise with Contextual RQ-Transformer can outperformprevious approaches for class- and text-conditional image generation.
Revised: In this section, we show that our Draft-and-Revise with Contextual RQ-Transformer can outperformprevious approaches for class- and text-conditional image generation.

Original: In addition, we conduct anextensive ablation study to understand the effects of Draft-and-Revise decoding on the quality anddiversity of generated images, and the sampling speed.
Revised: In addition, we conduct anextensive ablation study to understand the effects of Draft-and-Revise decoding on the quality anddiversity of generated images, and the sampling speed.

Original: We use the publicly released RQ-VAE [23]to represent a 256 × 256 resolution of images as 8 × 8 × 4 codes.
Revised: We use the publicly released RQ-VAE [23]to represent a 256 × 256 resolution of images as 8 × 8 × 4 codes.

Original: For a fair comparison, we make
Revised: For a fair comparison, we make

Original: Contextual RQ-Transformer have the same model size as the previous RQ-Transformer [23].
Revised: Contextual RQ-Transformer have the same model size as the previous RQ-Transformer [23].

Original: Fortraining, the quarter-period of cosine is used as the mask scheduling function γ in Eq.
Revised: Fortraining, the quarter-period of cosine is used as the mask scheduling function γ in Eq.

Original: 10 followingthe previous studies [6, 24].
Revised: 10 followingthe previous studies [6, 24].

Original: We include the implementation details in Appendix.
Revised: We include the implementation details in Appendix.

Original: 4.1 Class-conditional Image Generation
Revised: 4.1 Class-conditional Image Generation

Original: We train Contextual RQ-Transformer with 333M, 821M, and 1.4B parameters on ImageNet [7] forclass-conditional image generation.
Revised: We train Contextual RQ-Transformer with 333M, 821M, and 1.4B parameters on ImageNet [7] forclass-conditional image generation.

Original: For Draft-and-Revise decoding, we use T draft = 64 , T revise = 2 ,and M = 2 .
Revised: For Draft-and-Revise decoding, we use T draft = 64 , T revise = 2 ,and M = 2 .

Original: We use temperature scaling [19] and classifier-free guidance [20] only in the revise phase,while none of the strategies are applied in the draft phase.
Revised: We use temperature scaling [19] and classifier-free guidance [20] only in the revise phase,while none of the strategies are applied in the draft phase.

Original: Fréchet Inception Distance (FID) [18],
Revised: Fréchet Inception Distance (FID) [18],

Original: Inception Score (IS) [29], and Precision and Recall [22] are used for evaluation measures.
Revised: Inception Score (IS) [28], and Precision and Recall [22] are used for evaluation measures.

Original: Table 1 shows that Contextual RQ-Transformer significantly outperforms the previous approaches.
Revised: Table 1 shows that Contextual RQ-Transformer significantly outperforms the previous approaches.

Original: Notably, Contextual RQ-Transformer with 333M parameters outperforms RQ-Transformers with1.4B and 3.8B parameters on all evaluation measures, despite having only about 4.2 × and 11.4 ×fewer parameters.
Revised: Notably, Contextual RQ-Transformer with 333M parameters outperforms RQ-Transformers with1.4B and 3.8B parameters on all evaluation measures, despite having only about 4.2 × and 11.4 ×fewer parameters.

Original: In addition, the performance is improved as the number of parameters increases to821M and 1.4B.  Contextual RQ-Transformer can achieve the lower FID score without a pretrainedclassifier than ADM-G and 3.8B parameters of RQ-Transformer with the use of pretrained classifier.
Revised: In addition, the performance is improved as the number of parameters increases to821M and 1.4B. Our model with 333M parameters is competitive with MaskGIT [6], and hence weconduct additional analysis on the effect of confidence-based decoding in Appendix C. Contextual RQ-Transformer can achieve the lower FID score without a pretrained classifier than ADM-G and3.8B parameters of RQ-Transformer with the use of pretrained classifier.

Original: StyleGAN-XL also uses a pretrained classifier during both training and image generation and achievesthe lowest FID in Table 1.
Revised: StyleGAN-XL also usesa pretrained classifier during both training and image generation and achieves the lowest FID in Table 1.

Original: However, our model with 1.4B parameters has higher precision and recallthan StyleGAN-XL, implying that our model generates images of better fidelity and diversity withouta pretrained classifier.
Revised: However, our model with 1.4B parameters has higher precision and recall than StyleGANXL, implying that our model generates images of better fidelity and diversity without a pretrainedclassifier.

Original: Our high performance without a classifier is remarkable, since the gradientguidance and rejection sampling are the tailored techniques to the model-based evaluation metricsin Table 1.
Revised: Our high performance without a classifier is remarkable, since the gradient guidanceand rejection sampling are the tailored techniques to the model-based evaluation metrics in Table 1.

Original: Considering that the performance is marginally improved as the number of parametersincreases from 821M to 1.4B, an improved RQ-VAE can boost the performance of Contextual RQ-Transformer, since the reconstruction quality determines the best results of generated images.
Revised: Considering that the performance is marginally improved as the number of parameters increases from821M to 1.4B, an improved RQ-VAE can boost the performance of Contextual RQ-Transformer,since the reconstruction quality determines the best results of generated images.

Original: 4.2 Text-conditional Image Generation
Revised: 4.2 Text-conditional Image Generation

Original: [Table]
Revised: [Table]

Original: We train Contextual RQ-Transformer with333M and 654M parameters on CC-3M [33]for text-to-image (T2I) generation.
Revised: We train our model with 333M and 654Mparameters on CC-3M [32] for text-to-image(T2I) generation as described in Appendix A.

Original: We use Byte Pair Encoding [32, 36] to encode a textcondition into 32 tokens.
Revised: We use Byte Pair Encoding [31, 34] to encodea text condition into 32 tokens.

Original: We also report CLIPscore [26] with ViT-B/32 [11] to measure thecorrespondence between texts and images.
Revised: We also report CLIP-score [25] with ViT-B/32 [11] as thecorrespondence between texts and images.

Original: Contextual RQ-Transformer in Table 2 outperforms the previous T2I generation models.
Revised: Contextual RQ-Transformer in Table 2 outperforms the previous T2I generation models.

Original: Contextual RQ-Transformer with 333M parameters achieves better FID than RQ-Transformerwith 654M parameters, and outperforms ImageBART and LDM-4, although our model has 12 × fewer parameters than ImageBART.
Revised: Contextual RQ-Transformer with 333M parameters achieves better FID than RQ-Transformerwith 654M parameters, and outperforms ImageBART and LDM-4, although our model has 12 × fewer parameters than ImageBART.

Original: When weincrease the number of parameters to 654M, our model achieves state-of-the-art FID on CC-3M. 
Revised: When weincrease the number of parameters to 654M, our model achieves state-of-the-art FID on CC-3M. In

Original: Meanwhile, our model does not improve the CLIP score of RQ-Transformer, but achieves competitiveresults with fewer parameters.

Original: In Figure 3, our model generates images with unseen texts in CC-3M.[Figure]
Revised: Figure 3, our model generates images with unseen texts in CC-3M.

Original: T draft
Revised: T draft

Original: [Figure]

Original: F I D
Revised: F I D

Original: [Figure] (a) [Figure] [Figure]{+0 
+}
Revised: Draft only Draft + Revise (a) {+0 
+}

Original: [Figure] [Figure] P [Figure]
Revised: Number of revisions ( M ) 0.

Original:  ci s i o n [Figure] [Figure] [Figure]
Revised: P r e c

Original: c [Figure]
Revised: i s i o n 0.

Original:  Recall (b)     64 T [Figure] [Figure] [Figure]
Revised: R e c a ll Precision Recall (b)     64 T revise 2.

Original: [Figure]

Original: [Figure]

Original:  I D
Revised: F I D

Original: [Figure] (c)
Revised: Draft only Revise, M =1 Revise, M = (c)

Original: 4.3 Conditional Image Inpainting
Revised: 4.3 Conditional Image Inpainting

Original: We conduct conditional image inpainting where a model infills a masked area according to the givencondition and contexts.
Revised: We conduct conditional image inpainting where a model infills a masked area according to the givencondition and contexts.

Original: Figure 1 shows the example of image inpainting by RQ-Transformer (middle)and Contextual RQ-Transformer (right), when the class-condition is school bus .
Revised: Figure 1 shows the example of image inpainting by RQ-Transformer (middle)and Contextual RQ-Transformer (right), when the class-condition is school bus .

Original: RQ-Transformercannot attend to the right and bottom sides of the masked area and fails to generate a coherent imagewith given contexts.
Revised: RQ-Transformercannot attend to the right and bottom sides of the masked area and fails to generate a coherent imagewith given contexts.

Original: However, our model can complete the image to be coherent with given contextsby exploiting global contexts.
Revised: However, our model can complete the image to be coherent with given contextsby exploiting global contexts.

Original: We attach more examples of image inpainting in Appendix.
Revised: We attach more examples of image inpainting in Appendix.

Original: 4.4 Ablation Study on Draft-and-Revise
Revised: 4.4 Ablation Study on Draft-and-Revise

Original: We conduct an extensive ablation study to demonstrate the effectiveness of Draft-and-Revise decodingof our framework.
Revised: We conduct an extensive ablation study to demonstrate the effectiveness of Draft-and-Revise decodingof our framework.

Original: We use Contextual RQ-Transformer with 821M parameters trained on ImageNet.
Revised: We use Contextual RQ-Transformer with 821M parameters trained on ImageNet.

Original: Quality improvement of draft images in the revise phase Figure 4(a) shows the effects of T drafton draft images and their quality improvement in the revised phase with T revise = 2 and M = 2 .
Revised: Quality improvement of draft images in the revise phase Figure 4(a) shows the effects of T on draft images and their quality improvement in the revised phase with T revise = 2 and M = 2 .

Original: In the draft phase, FID is improved as T draft increases from 4 to 64.
Revised: In the draft phase, FID is improved as T draft increases from 4 to 64.

Original: At each inference, Contextual
Revised: At each inference, Contextual

Original: RQ-Transformer generates N/T draft code stacks in parallel, starting with the empty sequence.
Revised: RQ-Transformer generates N/T draft code stacks in parallel, starting with the empty sequence.

Original: Thus,the model with a large T draft generates a small number of code stacks at each inference and can avoidgenerating incoherent code stacks in the early stage of the draft phase.
Revised: Thus,the model with a large T draft generates a small number of code stacks at each inference and can avoidgenerating incoherent code stacks in the early stage of the draft phase.

Original: Although FIDs in the draftphase are worse, they are significantly improved in the revise phase as shown in Figure 5.
Revised: Although FIDs in the draftphase are worse due to low precision, we intend to increase the diversity of generated images, sincethe precision is significantly improved in the revise phase as shown in Figure 4(b).

Original: Effect of  M and  T revise in the revise phase Figure 4(b) shows the effects of the number of updates M in the revise phase on the quality and diversity of generated images.
Revised: Effect of revision iteration M and partition size T revise in the revise phase Figure 4(b) showsthe effects of the number of updates M in the revise phase on the quality and diversity of generatedimages.

Original: Since the quality-diversitytrade-off exists as the updates are repeated, we select M = 2 as the default hyperparameter to balancethe precision and recall, considering that the increase of precision starts to slow down.
Revised: Since the quality-diversity trade-off exists as the updates are repeated, we select M = 2 as thedefault hyperparameter to balance the precision and recall, considering that the increase of precisionstarts to slow down.

Original: Interestingly, Figure 5 shows that the overall contents remain unchanged even after M > 2 .
Revised: Interestingly, Figure 5 shows that the overall contents remain unchanged even after M > 2 .

Original: Thus, we claim that Draft-and-Revise decoding does not harm the perceptual diversity of generated images throughoutthe revise phase despite the consistent deterioration of recall.
Revised: Thus, we claim that Draft-and-Revise decoding does not harm the perceptual diversityof generated images throughout the revise phase despite the consistent deterioration of recall.

Original: Figure 4(c) shows the effects of T revise on the quality of generated images.
Revised: Figure 4(c) shows the effects of T revise on the quality of generated images.

Original: The FIDs are significantlyimproved in the revise phase regardless of the choice of T revise , but increasing T revise slightly deteriorates FIDs.
Revised: The FIDs are significantlyimproved in the revise phase regardless of the choice of T revise , but increasing T revise slightly deteriorates FIDs.

Original: We remark that some code stacks of a draft can be erroneous due to its low quality,and a model with large T revise slowly updates a small number of code stacks at once in the revisephase.
Revised: We remark that some code stacks of a draft can be erroneous due to its low quality,and a model with large T revise slowly updates a small number of code stacks at once in the revisephase.

Original: Therefore, the updates with large T revise can be more influenced by the erroneous code stacks.
Revised: Therefore, the updates with large T revise can be more influenced by the erroneous code stacks.

Original: Although T revise = 2 updates half of an image at once, our draft-and-revise decoding successfullyimproves the quality of generated images, while preserving the global contexts of drafts, as shownin Figure 5.
Revised: Although T revise = 2 updates half of an image at once, our draft-and-revise decoding successfullyimproves the quality of generated images, while preserving the global contexts of drafts, as shownin Figure 5.

Original: The study on self-supervised learning [17] also reports similar results, where a maskedauto-encoder reconstructs the global contexts of an image after masking half of the image.
Revised: The study on self-supervised learning [17] also reports similar results, where a maskedauto-encoder reconstructs the global contexts of an image after masking half of the image.

Original: [Table]
Revised: [Table]

Original: Quality-diversity control of Draft-and-Revise
Revised: Quality-diversity control of Draft-and-Revise

Original: Our
Revised: Our

Original: Draft-and-Revise decoding can effectively control thequality-diversity trade-off in generated images.
Revised: Draft-and-Revise decoding can effectively control thequality-diversity trade-off in generated images.

Original: Table 3shows FID, precision (P), and recall (R) according tothe use of classifier-free guidance [20] with a scale of1.8, while applying temperature scaling with 0.8 only tothe revise phase.
Revised: Table 3shows FID, precision (P), and recall (R) according tothe use of classifier-free guidance [20] with a scale of1.8, while applying temperature scaling with 0.8 only tothe revise phase.

Original: Contextual RQ-Transformer withoutthe guidance already outperforms RQ-Transformer with3.8B parameters and demonstrates the effectiveness of ourframework.
Revised: Contextual RQ-Transformer withoutthe guidance already outperforms RQ-Transformer with3.8B parameters and demonstrates the effectiveness of ourframework.

Original: When the guidance is used for both draft andrevise phases, the precision dramatically increases but the recall decreases to 0.33.
Revised: When the guidance is used for both draft andrevise phases, the precision dramatically increases but the recall decreases to 0.33.

Original: Consequently,
Revised: Consequently,

Original: FID becomes worse due to the lack of diversity in generated images.
Revised: FID becomes worse due to the lack of diversity in generated images.

Original: However, when the guidance isapplied only to the revise phase, our model achieves the lowest FID, as the quality and diversity arewell-balanced.
Revised: However, when the guidance isapplied only to the revise phase, our model achieves the lowest FID, as the quality and diversity arewell-balanced.

Original: Thus, the explicitly separated two phases of Draft-and-Revise can effectively controlthe issue of quality-diversity trade-off by generating diverse drafts and then improving their quality.
Revised: Thus, the explicitly separated two phases of Draft-and-Revise can effectively controlthe issue of quality-diversity trade-off by generating diverse drafts and then improving their quality.

Original: [Table]
Revised: [Table]

Original: Trade-off between quality and sampling speed
Revised: Trade-off between quality and sampling speed

Original: Afterwe fix T revise = 2 and M = 2 , the trade-off between FIDand the sampling speedis analyzed in Table 4 according to T draft  .
Revised: Table 4shows that our framework can control the trade-off betweenthe quality and sampling speed according to T draft after wefix T revise = 2 and M = 2 .

Original: Following the previous study [23], we generate 5,000samples with batch size of 100.
Revised: We generate 5,000 samples withbatch size of 100 as the previous study [23].

Original: Contextual RQ-Transformerwith T draft = 8 outperformsVQGAN and RQ-Transformerwith 1.4B parameters in terms of both FID  and the samplingspeed.
Revised: Contextual RQ-Transformer with T draft = 8 outperforms both FID andsampling speed of VQGAN and RQ-Transformer with 1.4Bparameters.

Original: Although the sampling speed becomes slow withincreased T draft , the FID scores are consistently improved.
Revised: Although the sampling speed becomes slowwith increased T draft , the FID scores are consistently improved.

Original: We remark that the sampling speed with T draft = 64 is about3 × slower than RQ-Transformer, but our model outperforms3.8B parameters of RQ-Transformer with rejection samplingin Table 1.
Revised: We remark that the sampling speed with T draft = 64is about 3 × slower than RQ-Transformer, but our modeloutperforms 3.8B parameters of RQ-Transformer with rejection sampling in Table 1.

Original: The results represent that our framework has inexpensivecomputational costs to generatehigh-quality images, since rejection sampling requires generating up to 20 × more samples than ours.
Revised: The results show that our framework has inexpensive costs to generatehigh-quality images, considering that rejection sampling requires generating up to 20 × more samples.

Original:  Conclusion
Revised:  Conclusion

Original: In this study, we have proposed Draft-and-Revise for an effective image generation framework with
Revised: In this study, we have proposed Draft-and-Revise for an effective image generation framework with

Original: Contextual RQ-Transformer.
Revised: Contextual RQ-Transformer.

Original: After an image is represented as a sequence of code stacks, Contextual
Revised: After an image is represented as a sequence of code stacks, Contextual

Original: RQ-Transformer is trained to infill a randomly masked sequence.
Revised: RQ-Transformer is trained to infill a randomly masked sequence.

Original: Then, Draft-and-Revise decoding isused to generate high-quality images by first generating a draft image with diverse contents and thenimproving its visual quality based on the global contexts of the draft.
Revised: Then, Draft-and-Revise decoding isused to generate high-quality images by first generating a draft image with diverse contents and thenimproving its visual quality based on the global contexts of the draft.

Original: Consequently, we can achievestate-of-the-art results on ImageNet and CC-3M, demonstrating the effectiveness of our framework.
Revised: Consequently, we can achievestate-of-the-art results on ImageNet and CC-3M, demonstrating the effectiveness of our framework.

Original: Our study has two main limitations to be further explored.
Revised: Our study has two main limitations to be further explored.

Original: Firstly, Draft-and-Revise decoding alwaysupdates all code stacks in the revise phase, although some code stacks might not need an update.
Revised: Firstly, Draft-and-Revise decoding alwaysupdates all code stacks in the revise phase, although some code stacks might not need an update.

Original: Infuture work, a selective method can be developed to improve the efficiency of the revise phase by asophisticated approach.
Revised: Infuture work, a selective method can be developed to improve the efficiency of the revise phase by asophisticated approach.

Original: Secondly, our generative model is not validated on various downstream tasks.
Revised: Secondly, our generative model is not validated on various downstream tasks.

Original: Since masked token modeling is successful self-supervised learning for texts [8] and images [3, 17], aunified model for both generative and discriminative tasks [21] is worth exploration for future work.
Revised: Since masked token modeling is successful self-supervised learning for texts [8] and images [3, 17], aunified model for both generative and discriminative tasks [21] is worth exploration for future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] Our code is
Revised: [Yes] Our code is

Original: included in the supplemental material.
Revised: included in the supplemental material.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See Sec. 4 and Appendix.
Revised: [Yes] See Sec. 4 and Appendix.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix.4.
Revised: [Yes] See Appendix.4.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We used publicly available datasets and cited them in Sec. 4.
Revised: [Yes] We used publiclyavailable datasets and cited them in Sec. 4.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]Our code is included in the supplemental material.
Revised: [Yes] Our code is included in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]