Original: CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers
Revised: CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

Original: Abstract
Revised: Abstract

Original: Development of transformer-based text-to-image models is impeded by its slowgeneration and complexity, for high-resolution images.
Revised: Development of transformer-based text-to-image models is impeded by its slow generation and complexity, for high-resolution images.

Original: In this work, we put forwarda solution based on hierarchical transformers and local parallel autoregressivegeneration.
Revised: In this work, we put forward a solution based on hierarchical transformers and local parallel autoregressive generation.

Original: We pretrain a 6B-parameter transformer with a simple and ﬂexibleself-supervised task, a cross-modal general language model (CogLM) and ﬁnetune it for fast super-resolution.
Revised: We pretrain a 6B-parameter transformer with a simple and ﬂexible self-supervised task, a cross-modal general language model (CogLM) and ﬁnetune it for fast super-resolution.

Original: The new text-to-image system, CogView2, showscompetitive generation performance to the concurrent state-of-the-art DALL-E-2,and naturally supports interactive text-guided editing on images.
Revised: The new text-to-image system, CogView2, shows competitive generation performance to the concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images

Original: A lion man is typing in the o ﬃ ce. A beautiful girl is hugging a husky. A lion teacher wearing a suit is in front of a blackboard. A robot is riding under the blue and cloudy sky.
Revised: [Figure]

Original: Several youths are talking in a bar. A young woman is taking photos. A tiger with angel’s wings. A girl holding an oil-paper umbrella in a rainy lane.
Revised: 36th Conference on Neural Information

Original: Earth in the Eye.

Original: A magniﬁcent church. Sketch. Mount Fuji, cherry blossom and Akita dog. Oil painting.
Revised: Systems (

Original: A pirate captain with a skull.

Original: Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: Do not distribute.

Original:  Introduction
Revised:  Introduction

Original: Recently, text-to-image generation has been greatly advanced by large-scale pretrained transformers,e.g. DALL-E [25] and CogView [3].
Revised: Recently, text-to-image generation has been greatly advanced by large-scale pretrained transformers, e.g. DALL-E [26] and CogView [3].

Original: These models learn to generate image tokens in an autoregressiveway.
Revised: These models learn to generate image tokens in an autoregressive way.

Original: However, they also suffer from the following disadvantages:
Revised: However, they also suffer from the following disadvantages:

Original: Slow generation.
Revised: Slow generation.

Original: Generation of autoregressive models usually is much slower than generationof non-autoregressive models, e.g. GANs [10], with the same FLOPs.
Revised: Generation of autoregressive models usually is much slower than generation of non-autoregressive models, e.g. GANs [10], with the same FLOPs.

Original: Instead of employing alarge number of parameters, this shortcoming is mainly attributed to the nature of token-by-tokengeneration used in the autoregressive models cannot exploit the parallel computing ability of GPUs,even after caching hidden states [24].
Revised: Instead of employing a large number of parameters, this shortcoming is mainly attributed to the nature of token-by-token generation used in the autoregressive models cannot exploit the parallel computing ability of GPUs, even after caching hidden states [25].

Original: This is a signiﬁcant limitation.
Revised: This is a signiﬁcant limitation.

Original: Expensive high-resolution training.
Revised: Expensive high-resolution training.

Original: The current large-scale pretrained models are generally basedon Transformers [29], where the attention operation has both time and space complexity of O ( n 2 )for training sequences of length n .
Revised: The current large-scale pretrained models are generally based on Transformers [30], where the attention operation has both time and space complexity of O ( n 2 ) for training sequences of length n .

Original: Within a limited budget, we face a trade-off between the numberof parameters, representing the modeling power, and the resolution of the generated images.
Revised: Within a limited budget, we face a trade-off between the number of parameters, representing the modeling power, and the resolution of the generated images.

Original: For thisreason, most current text-to-image models choose a resolution of 32 × 32 tokens (usually 256 × 256pixels) [3, 25, 11], which is far less dense than the resolution of the real photos.
Revised: For this reason, most current text-to-image models choose a resolution of 32 × 32 tokens (usually 256 ×pixels) [3, 26, 11], which is far less dense than the resolution of the real photos.

Original: Unidirectionality.
Revised: Unidirectionality.

Original: For images, autoregressive models, e.g. GPTs, usually generate tokens in rasterscan order.
Revised: For images, autoregressive models, e.g. GPTs, usually generate tokens in rasterscan order.

Original: This order shows the best perplexity during the evaluation [7].
Revised: This order shows the best perplexity during the evaluation [7].

Original: However, this ordermakes the models unaware of the tokens below or on the right side during generation, as a resulttext-guided inﬁlling is not supported.
Revised: However, this order makes the models unaware of the tokens below or on the right side during generation, as a result text-guided inﬁlling is not supported.

Original: Moreover, the unidirectionality leads to a gap between thepretrained text-to-image models and vision transformers (ViTs) [5] based on bidirectional maskedprediction, e.g. MAE [12] and SimMIM [32]—limiting their application on traditional visual tasks,such as image classiﬁcation and object detection.
Revised: Moreover, the unidirectionality leads to a gap between the pretrained text-to-image models and vision transformers (ViTs) [5] based on bidirectional masked prediction, e.g. MAE [12] and SimMIM [34]—limiting their application on traditional visual tasks, such as image classiﬁcation and object detection.

Original: Present Work.
Revised: Present Work.

Original: To overcome these defects, we ﬁrst propose a simple and versatile pretraining method,a C ross-M o dal g eneral L anguage M odel (CogLM).
Revised: To overcome these defects, we ﬁrst propose a simple and versatile pretraining method, a C ross-M o dal g eneral L anguage M odel (CogLM).

Original: Our CogLM masks various types of tokens inthe sequence of text and image tokens, and learns to predict them autoregressively.
Revised: Our CogLM masks various types of tokens in the sequence of text and image tokens, and learns to predict them autoregressively.

Original: Speciﬁcally, (1) ifwe mask all the image tokens, the task becomes the same as the original CogView [3] in performinga text-to-image generation task; (2) if we mask random patches of image tokens, it works similarly to
Revised: Speciﬁcally, (1) if we mask all the image tokens, the task becomes the same as the original CogView [3] in performing a text-to-image generation task; (2) if we mask random patches of image tokens, it works similarly to

Original: MAE as an inﬁlling task; (3) if we mask text tokens, the task becomes image captioning.
Revised: MAE as an inﬁlling task; (3) if we mask text tokens, the task becomes image captioning.

Original: The versatility of CogLM enables us to ﬁne-tune a pretrained CogLM for different downstream tasks,and constructs a hierarchical model, CogView2.There are three steps in the hierarchical generationprocess as follows:
Revised: The versatility of CogLM enables us to ﬁne-tune a pretrained CogLM for different downstream tasks, and constructs a hierarchical model, CogView2.There are three steps in the hierarchical generation process as follows:

Original: First, we generate a batch of low-resolution images ( 20 × 20 tokens in CogView2) using the pretrained CogLM and then (optionally) ﬁlter out the bad samples based on the perplexityof CogLM image captioning, which is the post-selection method introduced in CogView [3].2.
Revised: First, we generate a batch of low-resolution images ( 20 × 20 tokens in CogView2) using the pretrained CogLM, and then (optionally) ﬁlter out the bad samples based on the perplexity of CogLM image captioning, which is the post-selection method introduced in CogView [3].

Original: The generated images are mapped into 60 × 60 -token images by a direct super-resolutionmodule ﬁne-tuned from the pretrained CogLM.
Revised: The generated images are mapped into 60 × 60 -token images by a direct super-resolution module ﬁne-tuned from the pretrained CogLM.

Original: We use local attention implemented by ourcustomized CUDA kernel to reduce the training expense.
Revised: We use local attention implemented by our customized CUDA kernel to reduce the training expense.

Original: The high-resolution images fromthis step usually have inconsistent textures and lack details.3.
Revised: The high-resolution images from this step usually have inconsistent textures and lack details.

Original: These high-resolution images are reﬁned via another iterative super-resolution module ﬁnetuned from the pretrained CogLM.
Revised: These high-resolution images are reﬁned via another iterative super-resolution module ﬁnetuned from the pretrained CogLM.

Original: Most tokens are re-masked and re-generated in a localparallel autoregressive (LoPAR) way, which is much faster than the original autoregressivegeneration.
Revised: Most tokens are re-masked and re-generated in a local parallel autoregressive (LoPAR) way, which is much faster than the original autoregressive generation.

Original: How does CogView2 conquer the three defects?
Revised: How does CogView2 conquer the three defects?

Original: First, during pretraining the masked patch predictiontask trains CogLM to handle bidirectional context, making it easy to adapt to bidirectional tasks, suchas direct and iterative super-resolution.
Revised: First, during pretraining the masked patch prediction task trains CogLM to handle bidirectional context, making it easy to adapt to bidirectional tasks, such as direct and iterative super-resolution.

Original: Second, the hierarchical design allows us to care only aboutthe local coherence at a high-resolution level.
Revised: Second, the hierarchical design allows us to care only about the local coherence at a high-resolution level.

Original: In this way, the local attention can be leveraged toreduce the training expense.
Revised: In this way, the local attention can be leveraged to reduce the training expense.

Original: Third, the local parallel autoregressive generation can reduce modelrun times from 3,600 to 6 (1/600 only), signiﬁcantly accelerating the generation of high-resolutionimages.
Revised: Third, the local parallel autoregressive generation can reduce model run times from 3,600 to 6 (1/600 only), signiﬁcantly accelerating the generation of high-resolution images.

Original: CogView2 is about 10 × faster than the CogView (with sliding-window super-resolution) forgenerating images of similar resolution and better quality.
Revised: CogView2 is about 10 × faster than the CogView (with sliding-window super-resolution) for generating images of similar resolution and better quality.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Input Text:
Revised: Input Text:

Original: V Q V A
Revised: V Q V A

Original: E Discretize
Revised: E Discretize

Original: Image Tokenizer
Revised: Image Tokenizer

Original: [Table]
Revised: [Table]

Original: Input Image:
Revised: Input Image:

Original: [Table]
Revised: [Table]

Original: HOvzHTzkJbD4Qczrn3JvcECWdKO863VVpZXVvfKG9WtrZ3dvfs/YO2ilNJoUVjHstuQBRwJqClmebQTSSQKODQCcY3ud95AKlYLO71JAE/IkPBQkaJNlLfPvJiYweSUMi8kUry+9JJ9HTat6tOzZkBLxO3IFVUoNm3v7xBTNMIhKacKNVzzRw/I1IzymFa8VIFZv6YDKFnqCARKD+brTDFp0YZ4DCW5giNZ+rvjoxESk2iwFRGRI/UopeL/3m9VIfXfsZEkmoQdP5QmHKsY5zngQdMAtV8Ygihkpm [Figure] YjoiJg9tUquYENzFlZdJ+7zmXtScu3q1US [Figure]
Revised: HOvzHTzkJbD4Qczrn3JvcECWdKO863VVpZXVvfKG9WtrZ3dvfs/YO2ilNJoUVjHstuQBRwJqClmebQTSSQKODQCcY3ud95AKlYLO71JAE/IkPBQkaJNlLfPvJiYweSUMi8kUry+9JJ9HTat6tOzZkBLxO3IFVUoNm3v7xBTNMIhKacKNVzzRw/I1IzymFa8VIFZv6YDKFnqCARKD+brTDFp0YZ4DCW5giNZ+rvjoxESk2iwFRGRI/UopeL/3m9VIfXfsZEkmoQdP5QmHKsY5zngQdMAtV8Ygihkpm/YjoiJg9tUquYENzFlZdJ+7zmXtScu3q1US/iKKNjdILOkIuuUAPdoiZqIYoe0TN6RW/Wk/VivVsf89KSVfQcoj

Original: Wk

Original: [Figure]
Revised: [Figure]

Original: wPn8A712XuA
Revised: wPn8A712XuA

Original: [Figure]
Revised: [Figure]

Original:  Related Work
Revised:  Related Work

Original: [Figure] Text-to-image generation for arbitrary inputs is a long-held dream for many cross-modal machinelearning researchers.
Revised: Text-to-image generation for arbitrary inputs is a long-held dream for many cross-modal machinelearning researchers.

Original: Most early attempts to address this challenge were based on Generative
Revised: Most early attempts to address this challenge were based on Generative

Original: Adversarial Nets [10]; these include AttnGAN [33], DM-GAN [38], DF-GAN [27], et al.
Revised: Adversarial Nets [10]; these include AttnGAN [35], DM-GAN [40], DF-GAN [28], et al.

Original: Althoughthey can perform vivid synthesis on domain-speciﬁc datasets, such as Caltech-UCSD Birds 200,general-domain datasets, such as MS COCO [16], present great challenges for these methods.
Revised: Although they can perform vivid synthesis on domain-speciﬁc datasets, such as Caltech-UCSD Birds 200, general-domain datasets, such as MS COCO [17], present great challenges for these methods.

Original: DALLE [25], CogView [3] and similar works [31, 8] leverage VQ-VAE [28] to compress an image to asequence of discrete tokens and pretrain large transformers for autoregressive generation, greatlyimproving results in the general domain.
Revised: DALLE [26], CogView [3] and similar works [33, 8] leverage VQ-VAE [29] to compress an image to a sequence of discrete tokens and pretrain large transformers for autoregressive generation, greatly improving results in the general domain.

Original: LAFITE [37] learns to invert the pretrained CLIP [22]embeddings in the shared space of text and image for text-free training.
Revised: LAFITE [39] learns to invert the pretrained CLIP [23] embeddings in the shared space of text and image for text-free training.

Original: Recently, many researchershave turned to diffusion models, largely due to the slow generation defect of autoregressive models.
Revised: Recently, many researchers have turned to diffusion models, largely due to the slow generation defect of autoregressive models.

Original: One example is Glide [18].
Revised: One example is Glide [19].

Original: Non-autoregressive generation (NAR) is recently a popular topic in natural language generation—see Mask-Predict [9] and GLAT [20], which explores parallel decoding methods for autoregressivelike models.
Revised: Non-autoregressive generation (NAR) is recently a popular topic in natural language generation— see Mask-Predict [9] and GLAT [21], which explores parallel decoding methods for autoregressivelike models.

Original: Generation speed was not an issue in the era when GANs dominated the imagegeneration, but constitutes a considerable challenge for current autoregressive text-to-image models.
Revised: Generation speed was not an issue in the era when GANs dominated the image generation, but constitutes a considerable challenge for current autoregressive text-to-image models.

Original: M6-UFC [36] ﬁrst introduces NAR methods into the VQ-VAE framework, and similar ideas areadopted by VQ-diffusion [11] and MaskGIT [1].
Revised: M6-UFC [38] ﬁrst introduces NAR methods into the VQ-VAE framework, and similar ideas are adopted by VQ-diffusion [11] and MaskGIT [1].

Original: A possible drawback of pure NAR methods is thattokens sampled at the meantime might lead to global inconsistency in later steps during the generationof complex scenes.
Revised: A possible drawback of pure NAR methods is that tokens sampled at the meantime might lead to global inconsistency in later steps during the generation of complex scenes.

Original: Our method introduces a hierarchical design to combine the consistency merit ofautoregressive models and the speed advantage of NAR methods.
Revised: Our method introduces a hierarchical design to combine the consistency merit of autoregressive models and the speed advantage of NAR methods.

Original:  Method 
Revised:  Method 3.1 The Cross-Modal General Language Model

Original: 3.1 The Cross-Modal General Language Model

Original: While previous self-supervised pretext tasks often target at mask prediction in the computer vision [32,12], our approach pursues a uniﬁcation of autoregressive generation and bidirectional context-awaremask prediction.
Revised: While previous self-supervised pretext tasks often target at mask prediction in the computer vision [34, 12], our approach pursues a uniﬁcation of autoregressive generation and bidirectional context-aware mask prediction.

Original: In NLP, the General Language Model (GLM)
Revised: In NLP, the General Language Model (GLM)

Original: [6] suggests changing the direct mask predictioninto blockwise autoregressive generation.
Revised: [6] suggests changing the direct mask prediction into blockwise autoregressive generation.

Original: However, directly applying it to images would result inredundancy.
Revised: However, directly applying it to images would result in redundancy.

Original: For instance, the sizes of the masked image patches are ﬁxed, thus we do not need thecapacity of ﬁlling blocks of indeﬁnite length as in NLP.
Revised: For instance, the sizes of the masked image patches are ﬁxed, thus we do not need the capacity of ﬁlling blocks of indeﬁnite length as in NLP.

Original: Moreover, GLM inserts a sentinel token for each mask region to predict its ﬁrst token, which greatly increases the sequence length thus restrictsthe usage of 2D local attention.
Revised: Moreover, GLM inserts a sentinel token for each mask region to predict its ﬁrst token, which greatly increases the sequence length thus restricts the usage of 2D local attention.

Original: Based on the analysis above, we present a simpler and more general language model for both textand image data—Cross-modal general Language Model (CogLM).
Revised: Based on the analysis above, we present a simpler and more general language model for both text and image data—Cross-modal general Language Model (CogLM).

Original: As shown in Figure 2, CogLMtakes as input a concatenation of text and images tokenized by icetk  (See § 3.2), whose dictionarycontains 20,000 image tokens and 130,000 text (both Chinese and English) tokens.
Revised: As shown in Figure 2, CogLM takes as input a concatenation of text and images tokenized by icetk 1 (See § 3.2), whose dictionary contains 20,000 image tokens and 130,000 text (both Chinese and English) tokens.

Original: Formally, lett =
Revised: Formally, let t =

Original: [ t 1 , ..., t M ] be the text tokens
Revised: [ t 1 , ..., t M ] be the text tokens

Original: and im =
Revised: and im =

Original: [ im 1 , ..., im N 2 ] be the image tokens, where M and N 2are the lengths of text and image tokens respectively.
Revised: [ im 1 , ..., im N 2 ] be the image tokens, where M and Nare the lengths of text and image tokens respectively.

Original: The crucial step in CogLM is to sample k mask regions R = { [ l 0 , r 0 ] , ..., [ l k , r k ] } according tovarious strategies.
Revised: The crucial step in CogLM is to sample k mask regions R = { [ l 0 , r 0 ] , ..., [ l k , r k ] } according to various strategies.

Original: In practice, the following two strategies are used:
Revised: In practice, the following two strategies are used:

Original: • (Text-to-Image GPT)
Revised: • (Text-to-Image GPT)

Original: The input sequence is x =
Revised: The input sequence is x =

Original: [ t [BOI] im ] .
Revised: [ t [BOI] im ] .

Original: We mask all the image tokens, which is similar to the pretraining task of CogView [3].
Revised: We mask all the image tokens, which is similar to the pretraining task of CogView [3].

Original: • (A Combination of Mask Prediction and Image Captioning)
Revised: • (A Combination of Mask Prediction and Image Captioning)

Original: The input sequence is x =
Revised: The input sequence is x =

Original: [ im 0 ...{+
+}
Revised: [ im {+
+}

Original: im i ...
Revised: im i ...

Original: im j ...
Revised: im j ...

Original: im N 2 [BOE/C] t ] , where [BOE],[BOC] are separators meaningbeginning-of-English and beginning-of-Chinese used for the corresponding language.
Revised: im N 2 [BOE/C] t ] , where [BOE],[BOC] are separators meaning beginning-of-English and beginning-of-Chinese used for the corresponding language.

Original: wemask random patches and the text tokens.
Revised: we mask random patches and the text tokens.

Original: Ideally, the two tasks should be separated; but we combine them together for training efﬁciency.
Revised: Ideally, the two tasks should be separated; but we combine them together for training efﬁciency.

Original: Instead of replacing the tokens in the mask regions as [MASK] , we make no change in the input butbuild an attention mask A based on the mask regions.
Revised: Instead of replacing the tokens in the mask regions as [MASK] , we make no change in the input but build an attention mask A based on the mask regions.

Original: All tokens outside mask regions are seen ascontext and can be attended to by all other tokens.
Revised: All tokens outside mask regions are seen as context and can be attended to by all other tokens.

Original: A token in mask regions can only be attended toby the tokens in mask regions and behind it.
Revised: A token in mask regions can only be attended to by the tokens in mask regions and behind it.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: A [ i, [Equation] = [Equation]
Revised: A [ i, j ] =   1 , if ∀ [ l u , r u ] ∈ R,

Original: [Equation] l [Equation] [Equation]
Revised: j / ∈

Original:  1 , if j ≤ i and ∃ u, v (indices) , i ∈ [ l u , r u ] ∈ R, j ∈ [ l v , r v ] ∈ R, 0 , else.
Revised: [ l u , r u ] , 1 , if j ≤ i and ∃ u, v (indices) , i ∈ [ l u , r u ] ∈ R, j ∈ [ l v , r v ] ∈ R, 0 , else.

Original: [Equation]
Revised: [Equation]

Original: Figure 2 shows an example of the attention mask matrix of two mask regions.
Revised: Figure 2 shows an example of the attention mask matrix of two mask regions.

Original: In the mask regions, the model learns to predict the next token.
Revised: In the mask regions, the model learns to predict the next token.

Original: The loss function can be written asfollows:
Revised: The loss function can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: A fox is siting on the books
Revised: A fox is siting on the books.

Original: [Equation]

Original: where the x context denotes the tokens outside the mask regions.
Revised: where the x context denotes the tokens outside the mask regions.

Original: Inﬁlling.
Revised: Inﬁlling.

Original: Note that the ﬁrst token in each mask region is not predicted during training.
Revised: Note that the ﬁrst token in each mask region is not predicted during training.

Original: This feature seems to disable CogLM fromimage inﬁlling or cloze ﬁlling in natural language, but this problemactually has a simple solution.
Revised: This feature seems to disable CogLM from image inﬁlling or cloze ﬁlling in natural language, but this problem actually has a simple solution.

Original: During inference, we can move thelast context token before each mask region into it, as illustrated in
Revised: During inference, we can move the last context token before each mask region into it, as illustrated in

Original: Figure 3.
Revised: Figure 3.

Original: Although these moved tokens becomes blind spots formask regions before them, they have few negative effects in practice.
Revised: Although these moved tokens becomes blind spots for mask regions before them, they have few negative effects in practice.

Original: To further avoid this minor inﬂuence and fully maintain the contextinformation, we deal with each mask region individually.
Revised: To further avoid this minor inﬂuence and fully maintain the context information, we deal with each mask region individually.

Original: For eachregion, we move only the last context token before this region, andkeep all the known tokens outside the mask regions.
Revised: For each region, we move only the last context token before this region, and keep all the known tokens outside the mask regions.

Original: Thus, we cannot use the cached hidden states from the previous region, slightlyslowing down the multi-region inﬁlling.
Revised: Thus, we cannot use the cached hidden states from the previous region, slightly slowing down the multi-region inﬁlling.

Revised: See Appendix A for samples.

Original: Advantages over GPT [21], GLM [6] and MAE [12].
Revised: Advantages over GPT [22], GLM [6] and MAE [12].

Original: (GPT)
Revised: (GPT)

Original: The main advantage over GPT is that the modeling of bidirectionalcontexts is considered in CogLM, which will beneﬁt many tasks relying on global information,e.g. super-resolution in the next section and image classiﬁcation.
Revised: The main advantage over GPT is that the modeling of bidirectional contexts is considered in CogLM, which will beneﬁt many tasks relying on global information, e.g. super-resolution in the next section and image classiﬁcation.

Original: The importance of bidirectionalcontext has been veriﬁed in the comparison of BERT [2] and GPT on GLUE [30].
Revised: The importance of bidirectional context has been veriﬁed in the comparison of BERT [2] and GPT on GLUE [31].

Original: (GLM)
Revised: (GLM)

Original: The mainadvantage over GLM is simplicity.
Revised: The main advantage over GLM is simplicity.

Original: To unify the generation and bidirectional understanding, GLM needs to deﬁne many new special tokens and a new type of position embedding, insert a sentinel foreach mask region and change the order of input tokens.
Revised: To unify the generation and bidirectional understanding, GLM needs to deﬁne many new special tokens and a new type of position embedding, insert a sentinel for each mask region and change the order of input tokens.

Original: It destroys the spatial relevance in the imagedata and excludes the possibility of using 2D local attention or convolution.
Revised: It destroys the spatial relevance in the image data and excludes the possibility of using 2D local attention or convolution.

Original: (MAE) MAE is designedfor self-supervised learning on pure image data and is not ready for generation.
Revised: (MAE) MAE is designed for self-supervised learning on pure image data and is not ready for generation.

Original: Even without text, CogLM is more parameter-efﬁcient, because MAE is an encoder-decoder structure.
Revised: Even without text, CogLM is more parameter-efﬁcient, because MAE is an encoder-decoder structure.

Original: A considerablepart of parameters in encoders and decoders are learned for the same function, e.g. extracting basicfeatures from inputs.
Revised: A considerable part of parameters in encoders and decoders are learned for the same function, e.g. extracting basic features from inputs.

Original: 3.2 Pretraining
Revised: 3.2 Pretraining

Original: As we have introduced CogLM as a general pretraining framework, in this section, we will describethe details and hyperparameters of our pretrained CogLM.
Revised: As we have introduced CogLM as a general pretraining framework, in this section, we will describe the details and hyperparameters of our pretrained CogLM.

Original: Tokenization.
Revised: Tokenization.

Original: We have developed a uniﬁed tokenizer icetk of I mage, C hinese and E nglish.
Revised: We have developed a uniﬁed tokenizer icetk of I mage, C hinese and E nglish.

Original: As shownin DebertaV2 [13], a large vocabulary (128,000 tokens) offers many beneﬁts.
Revised: As shown in DebertaV2 [13], a large vocabulary (128,000 tokens) offers many beneﬁts.

Original: For text, we extracta bilingual vocabulary of 130,000 tokens in icetk and explicitly classify them as Chinese, English,
Revised: For text, we extract a bilingual vocabulary of 130,000 tokens in icetk and explicitly classify them as Chinese, English,

Original: Common or Rare Symbols, so that we can specify the generated language via a sampling mask.
Revised: Common or Rare Symbols, so that we can specify the generated language via a sampling mask.

Original: The image tokenizer is a 20,000-token ﬁrst-stage VQ-VAE [28], largely following the tokenizer in
Revised: The image tokenizer is a 20,000-token ﬁrst-stage VQ-VAE [29], largely following the tokenizer in

Original: CogView [3].
Revised: CogView [3].

Original: Inspired by Esser et al.
Revised: Inspired by Esser et al.

Original: [7], a term of perceptual loss [35] is added to the reconstructionloss, signiﬁcantly improving reconstruction performance.
Revised: [7], a term of perceptual loss [37] is added to the reconstruction loss, signiﬁcantly improving reconstruction performance.

Original: (See Appendix for details.)
Revised: (See Appendix for details.)

Original: Transformer.
Revised: Transformer.

Original: The backbone of our pretrained CogLM is a Transformer with Sandwich LayerNorm [3].
Revised: The backbone of our pretrained CogLM is a Transformer with Sandwich LayerNorm [3].

Original: The model has 6 billion parameters (48 layers, hidden size 3072, 48 attention heads),trained for 300,000 iterations in FP16 with batch size 4,096.
Revised: The model has 6 billion parameters (48 layers, hidden size 3072, 48 attention heads), trained for 300,000 iterations in FP16 with batch size 4,096.

Original: The sequence length is 512, consistingof 400 image tokens, 1 separator and up to 111 text tokens.
Revised: The sequence length is 512, consisting of 400 image tokens, 1 separator and up to 111 text tokens.

Original: Masking Strategy.
Revised: Masking Strategy.

Original: We randomly select a sampling strategy for each training sample.
Revised: We randomly select a sampling strategy for each training sample.

Original: For the maskprediction strategy, the analysis from SimMIM [32] exhibits the great importance of mask percentageand patch distribution.
Revised: For the mask prediction strategy, the analysis from SimMIM [34] exhibits the great importance of mask percentage and patch distribution.

Original: We follow their results to sample 4 × 4 token patches at random until 75% ofthe tokens are in the mask regions.
Revised: We follow their results to sample 4 × 4 token patches at random until 75% of the tokens are in the mask regions.

Original: For bilingual samples, we randomly choose one of the languagesduring training.
Revised: For bilingual samples, we randomly choose one of the languages during training.

Original: 3.3 Hierarchical Generation
Revised: 3.3 Hierarchical Generation

Original: Although the pretrained CogLM can generate images from text, the resolution is only 20 × 20 tokens( 160 × 160 pixels).
Revised: Although the pretrained CogLM can generate images from text, the resolution is only 20 × 20 tokens ( 160 × 160 pixels).

Original: The short sequence is intentional, for fast generation.
Revised: The short sequence is intentional, for fast generation.

Original: The versatility of CogLMallows us to ﬁne-tune it into super-resolution models.
Revised: The versatility of CogLM allows us to ﬁne-tune it into super-resolution models.

Original: The whole hierarchical pipeline makes up our
Revised: The whole hierarchical pipeline makes up our

Original: CogView2 system.
Revised: CogView2 system.

Original: Direct super-
Revised: Direct super-

Original: resolution.
Revised: resolution.

Original: In this step, we want a model to map a generated low-resolution imagetoken sequence im 0 ∈ [0 , 20000) 20 × 20 to a higher-resolution sequence im 1 ∈ [0 , 20000) 60 × 60 .
Revised: In this step, we want a model to map a generated low-resolution image token sequence im 0 ∈ [0 , 20000) 20 × 20 to a higher-resolution sequence im 1 ∈ [0 , 20000) 60 × 60 .

Original: Weﬁne-tune the pretrained CogLM into an encoder-decoder architecture.
Revised: We ﬁne-tune the pretrained CogLM into an encoder-decoder architecture.

Original: The input of the encoder is the20 × 20 sequence of generated image tokens, and the input of the decoder is just a 60 × 60 sequenceof [MASK] .
Revised: The input of the encoder is the 20 × 20 sequence of generated image tokens, and the input of the decoder is just a 60 × 60 sequence of [MASK] .

Original: We do not follow the original transformer [29] to add a cross-attention layer, instead wemake the tokens in the decoder attend both local tokens in decoder and encoder.
Revised: We do not follow the original transformer [30] to add a cross-attention layer, instead we make the tokens in the decoder attend both local tokens in decoder and encoder.

Original: This cross-resolutionlocal attention is implemented via a customized CUDA kernel introduced in section 4.2.
Revised: This cross-resolution local attention is implemented via a customized CUDA kernel introduced in section 4.2.

Original: Bothencoder and decoder are initialized using the pretrained CogLM.
Revised: Both encoder and decoder are initialized using the pretrained CogLM.

Original: In practice, we ﬁnd it enough toonly ﬁne-tune the weights of the attention layers in the decoder, so that we can ﬁx and share the otherparameters between the encoder and decoder to reduce the memory consumption.
Revised: In practice, we ﬁnd it enough to only ﬁne-tune the weights of the attention layers in the decoder, so that we can ﬁx and share the other parameters between the encoder and decoder to reduce the memory consumption.

Original: Although direct mapping is a traditional practice for super-resolution—e.g. SRCNN [4]—it is hardlyqualiﬁed as generation; it focuses more on texture transformation.
Revised: Although direct mapping is a traditional practice for super-resolution—e.g. SRCNN [4]—it is hardly qualiﬁed as generation; it focuses more on texture transformation.

Original: The loss function of direct mappingis token-based or pixel-based (MAE), meaning that it predicts or maximizes the marginal distributionp ( im 1 i | im 0 ) for each token i instead of p ( im 1 | im 0 ) .
Revised: The loss function of direct mapping is token-based or pixel-based (MAE), meaning that it predicts or maximizes the marginal distribution p ( im 1 i | im 0 ) for each token i instead of p ( im 1 | im 0 ) .

Original: As we use cross-entropy loss and a multinomialsampling during generation, we get
Revised: As we use cross-entropy loss and a multinomial sampling during generation, we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, we need to reﬁne im 1 using another module.

Original: [Figure]
Revised: [Figure]

Original: Iterative super-resolution.
Revised: Iterative super-resolution.

Original: In this step, we aim to reﬁne the initial high-resolution sequence im 1into a better one im 2 .
Revised: In this step, we aim to reﬁne the initial high-resolution sequence iminto a better one im 2 .

Original: The working principle of the reﬁnement is to break the independence of thegenerated tokens, while keeping the parallelism.
Revised: The working principle of the reﬁnement is to break the independence of the generated tokens, while keeping the parallelism.

Original: Thus, we propose a local parallel autoregressive(LoPAR) approach.
Revised: Thus, we propose a local parallel autoregressive (LoPAR) approach.

Original: The motivation of LoPAR is that the hierarchical process frees us from global dependence.
Revised: The motivation of LoPAR is that the hierarchical process frees us from global dependence.

Original: As longas we maintain 25% – a ratio from MAE [12] – random tokens as context, it is sufﬁcient to recoverthe global scene of the image.
Revised: As long as we maintain 25% – a ratio from MAE [12] – random tokens as context, it is sufﬁcient to recover the global scene of the image.

Original: If the re-generated tokens are coherent locally with 25% kept tokens,global coherence is also guaranteed.
Revised: If the re-generated tokens are coherent locally with 25% kept tokens, global coherence is also guaranteed.

Original: We mask 75% of the tokens of im 1 and assume that there is alocal window size σ ,
Revised: We mask 75% of the tokens of im 1 and assume that there is a local window size σ ,

Original: [Equation]
Revised: [Equation]

Original:  not masked. 
Revised: j is not masked. }

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 5) so that local attention is sufﬁcient and tokens from different local windows can be generated inparallel.
Revised: so that local attention is sufﬁcient and tokens from different local windows can be generated in parallel.

Original: To further increase the parallelism, we ﬁnd the local inconsistency usually occurs whendirectly adjacent (vertically or horizontally) tokens are generated at the same time.
Revised: To further increase the parallelism, we ﬁnd the local inconsistency usually occurs when directly adjacent (vertically or horizontally) tokens are generated at the same time.

Original: We factorize thegeneration process into different iterations diagonally as in Figure 4 and below: 
Revised: We factorize the generation process into different iterations diagonally as in Figure 4 and below: [Equation]

Revised: (cid:0) row

Original: [Equation]
Revised: [Equation]

Original:  where row ( i ) = (cid:98) i − 1 60 (cid:99) mod σ and col ( i ) 
Revised: i [Equation] where row ( i ) = (cid:98) i − 1 60 (cid:99) mod σ and col ( i ) =

Original: =

Original: ( i − 1) mod σ are the indices of row and column in thelocal window.
Revised: ( i − 1) mod σ are the indices of row and column in the local window.

Original: To implement the iterative super-resolution module, we ﬁne-tune the pretrained CogLM for 20,000iterations into a BERT-style masked prediction model on 60 × 60 -token sequences with local attention.
Revised: To implement the iterative super-resolution module, we ﬁne-tune the pretrained CogLM for 20, iterations into a BERT-style masked prediction model on 60 × 60 -token sequences with local attention.

Original: The mask ratio is sampled from { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 0 . 9 } for each sample.
Revised: The mask ratio is sampled from { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 0 . 9 } for each sample.

Original: During inference, we setthe local window size to σ = 6 and compress the iterative process from 2 σ − 1 to 6 iterations byarranging the unmasked tokens and merging the ﬁrst and ﬁnal iterations 1 .
Revised: During inference, we set the local window size to σ = 6 and compress the iterative process from 2 σ − 1 to 6 iterations by arranging the unmasked tokens and merging the ﬁrst and ﬁnal iterations 2 .

Original:  Plug-in Improved Techniques for Transformers4.1 Cluster Sampling
Revised:  Plug-in Improved Techniques for Transformers 4.1 Cluster Sampling

Original: [Figure]

Original: In autoregressive generation, the sampling strategy over the predicted distribution of the tokens iscrucial.
Revised: In autoregressive generation, the sampling strategy over the predicted distribution of the tokens is crucial.

Original: Top-k or top-p (nucleus) sampling [14] are the most common strategies, but suffer from anincomplete truncation problem.
Revised: Top-k or top-p (nucleus) sampling [14] are the most common strategies, but suffer from an incomplete truncation problem.

Original: [Figure]
Revised: [Figure]

Original: The vocabulary of the image tokens is learned by VQVAE [28],where the embeddings of some tokens are very similar.
Revised: The vocabulary of the image tokens is learned by VQVAE [29], where the embeddings of some tokens are very similar.

Original: Torepresent the frequent patterns at a ﬁner granularity, we use alarge vocabulary of 20,000 tokens, three times larger than thatof the previous works [25, 3], further exacerbating the situation.
Revised: To represent the frequent patterns at a ﬁner granularity, we use a large vocabulary of 20,000 tokens, three times larger than that of the previous works [26, 3], further exacerbating the situation.

Original: For instance, there are about 42 tokens basically “white” inicetk , which show subtle differences only when connected tosome other tokens.
Revised: For instance, there are about 42 tokens basically “white” in icetk , which show subtle differences only when connected to some other tokens.

Original: Although the sum of the probabilities ofthese “white” tokens might be large enough, most of them couldbe ﬁltered by top-k sampling.
Revised: Although the sum of the probabilities of these “white” tokens might be large enough, most of them could be ﬁltered by top-k sampling.

Original: Figure 5 illustrates the problem.
Revised: Figure 5 illustrates the problem.

Original: To solve the incomplete sampling problem, we propose clustersampling.
Revised: To solve the incomplete sampling problem, we propose cluster sampling.

Original: We group the 20,000 tokens into 500 clusters via Kmeans [17] based on their vectors in VQVAE.
Revised: We group the 20,000 tokens into 500 clusters via Kmeans [18] based on their vectors in VQVAE.

Original: During sampling,we ﬁrst sample a cluster using top-k sampling based on the sumof probabilities of tokens in the clusters, and then sample in thecluster.
Revised: During sampling, we ﬁrst sample a cluster using top-k sampling based on the sum of probabilities of tokens in the clusters, and then sample in the cluster.

Original: All the tokens within a cluster are treated as a wholeand will be ﬁltered or kept together, alleviating the incomplete truncation problem.
Revised: All the tokens within a cluster are treated as a whole and will be ﬁltered or kept together, alleviating the incomplete truncation problem.

Original: 4.2 Local Attention
Revised: 4.2 Local Attention

Original: The hidden size in the benchmark is 64.
Revised: The hidden size in the benchmark is 64.

Original: Locality is one of the most important properties of imagedata.
Revised: Locality is one of the most important properties of image data.

Original: Local operations, e.g. convolution, dominated thevisual computing before ViTs [5].
Revised: Local operations, e.g. convolution, dominated the visual computing before ViTs [5].

Original: Even attention in the
Revised: Even attention in the

Original: ViTs mainly deals with the interactions between local tokens [23].
Revised: ViTs mainly deals with the interactions between local tokens [24].

Original: We ﬁnd it possible to ﬁne-tune the pretrained
Revised: We ﬁnd it possible to ﬁne-tune the pretrained

Original: CogLM using local attention and textual attention, whichis generally compatible with the global attention weightsfrom pretraining.
Revised: CogLM using local attention and textual attention, which is generally compatible with the global attention weights from pretraining.

Original: However, 2D local attention cannot beimplemented efﬁciently using high-level framework, e.g.
Revised: However, 2D local attention cannot be implemented efﬁciently using high-level framework, e.g.

Original: Pytorch [19].
Revised: Pytorch [20].

Original: We develop a customized CUDA kernelto support both 2D local attention, 2D autoregressive local attention and cross-resolution local attention.
Revised: We develop a customized CUDA kernel to support both 2D local attention, 2D autoregressive local attention and cross-resolution local attention.

Original: In the
Revised: In the

Original: CUDA kernel implementation, we can save half of thecomputation in the matrix multiplication and do not needa causal attention mask for the autoregressive attention.
Revised: CUDA kernel implementation, we can save half of the computation in the matrix multiplication and do not need a causal attention mask for the autoregressive attention.

Original: Inthe super-resolution modules, we use local attention withthe receptive ﬁeld (RF) of 9 × 9 .
Revised: In the super-resolution modules, we use local attention with the receptive ﬁeld (RF) of 9 × 9 .

Original: Figure 6 show the benchmark for a single-head attention with hidden size 64 ona A100 GPU.
Revised: Figure 6 show the benchmark for a single-head attention with hidden size 64 on a A100 GPU.

Original: The advantage of our method will be moreobvious in autoregressive scenarios, which is up to 40 ×faster and consumes 1% memory than global attention on4,096 sequences.
Revised: The advantage of our method will be more obvious in autoregressive scenarios, which is up to 40 × faster and consumes 1% memory than global attention on 4,096 sequences.

Original: 4.3 Upweighting Textual Attention
Revised: 4.3 Upweighting Textual Attention

Original: Most text-image pairs are weakly relevant in the large training data of CogLM.
Revised: Most text-image pairs are weakly relevant in the large training data of CogLM.

Original: Even the modelperfectly ﬁts the data, it should have a considerable probability to generate irrelevant images.
Revised: Even the model perfectly ﬁts the data, it should have a considerable probability to generate irrelevant images.

Original: Tostrengthen the relevance, we leverage the explainability of the attention operation.
Revised: To strengthen the relevance, we leverage the explainability of the attention operation.

Original: We add a constant cto the attention scores from any token to the text tokens: (The attention mask is omitted for simplicity)
Revised: We add a constant c to the attention scores from any token to the text tokens: (The attention mask is omitted for simplicity)

Original: [Equation]
Revised: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) text part 0 ...(cid:124) (cid:123)(cid:122) (cid:125) image part ])
Revised: (cid:124) (cid:123)(cid:122) (cid:125) text part 0 ...(cid:124) (cid:123)(cid:122) (cid:125) image part ])

Original: V.
Revised: V.

Original: This technique costs ignorable time consumption but largely improves the textual relevance of thegenerated images.
Revised: This technique costs ignorable time consumption but largely improves the textual relevance of the generated images.

Original: In practice, c < 3 will not inﬂuence the quality of the images.
Revised: In practice, c < 3 will not inﬂuence the quality of the images.

Original:  Experiments 
Revised:  Experiments 5.1 Dataset

Original: 5.1 Dataset

Original: Our dataset for pretraining contains about 30 million text-image pairs, mostly overlapping with thatof CogView [3].
Revised: Our dataset for pretraining contains about 30 million text-image pairs, mostly overlapping with that of CogView [3].

Original: We ﬁlter about 5 million text-image pairs from the CogView dataset with somekeywords, e.g. “abstract” and “texture”, because they are mostly background images used for design.
Revised: We ﬁlter about 5 million text-image pairs from the CogView dataset with some keywords, e.g. “abstract” and “texture”, because they are mostly background images used for design.

Original: These images consist of repeating patterns and contribute little to text-to-image generation.
Revised: These images consist of repeating patterns and contribute little to text-to-image generation.

Original: We thenreplenish the dataset with 5 million tag-image pairs.
Revised: We then replenish the dataset with 5 million tag-image pairs.

Original: About half the text is translated from English,and both Chinese and English text are kept to train our bilingual CogLM.
Revised: About half the text is translated from English, and both Chinese and English text are kept to train our bilingual CogLM.

Original: Only the images whoseresolution is at least 480 × 480 are used to train the super-resolution modules.
Revised: Only the images whose resolution is at least 480 × 480 are used to train the super-resolution modules.

Original: 5.2 Machine Evaluation
Revised: 5.2 Machine Evaluation

Original: To compare with previous and concurrent works, we follow the most popular benchmark originatedfrom DALL-E [25], Fréchet Inception Distances and Inception Scores evaluated on MS-COCO [16].30,000 captions from the validation set are sampled to evaluate the FID.
Revised: To compare with previous and concurrent works, we follow the most popular benchmark originated from DALL-E [26], Fréchet Inception Distances and Inception Scores evaluated on MS-COCO [17]. 30,000 captions from the validation set are sampled to evaluate the FID.

Original: Since each image in COCOhas up to 5 different captions, we carefully select the sampled captions to describe different images.
Revised: Since each image in COCO has up to 5 different captions, we carefully select the sampled captions to describe different images.

Original: We generate 16 samples for each caption (translated into Chinese), and select the best one with thelowest caption perplexity (the Caption Score in [3]).
Revised: We generate 16 samples for each caption (translated into Chinese), and select the best one with the lowest caption perplexity (the Caption Score in [3]).

Original: Note that FID is not the perfect metric to evaluate
Revised: Note that FID is not the perfect metric to evaluate

Original: CogView2 because (1) the advantage of CogView2 is to generate high-resolution images, but weneed to resize the images back to 256 × 256 for meaningful comparison.
Revised: CogView2 because (1) the advantage of CogView2 is to generate high-resolution images, but we need to resize the images back to 256 × 256 for meaningful comparison.

Original: (2) There are mistakes whentranslating English captions into Chinese.
Revised: (2) There are mistakes when translating English captions into Chinese.

Original: (3) Our training data contain many single-object images,which are quite different from those in the distribution of COCO (common objects in context ).
Revised: (3) Our training data contain many single-object images, which are quite different from those in the distribution of COCO (common objects in context ) [Table]

Original: [Table]
Revised: [Table]

Original: The results of machine evaluation are demonstrated in Table 1.
Revised: The results of machine evaluation are demonstrated in Table 1.

Original: We ﬁnd that ﬁne-tuning CogLMon the MS-COCO dataset will largely improve the FID.
Revised: We ﬁnd that ﬁne-tuning CogLM on the MS-COCO dataset will largely improve the FID.

Original: During our ﬁne-tuning, FID diminishesfrom 24.0 (0 iteration) → 19.2 (2,500 iterations) → 17.5 (7,500 iterations).
Revised: During our ﬁne-tuning, FID diminishes from 24.0 (0 iteration) → 19.2 (2,500 iterations) → 17.5 (7,500 iterations).

Original: However, we ﬁnd thatthe quality (human evaluation) of generation deteriorates.
Revised: However, we ﬁnd that the quality (human evaluation) of generation deteriorates.

Original: Though the style is similar to COCO, thegeneration is not as accurate as for the non-ﬁne-tuned version, which also corresponds to the scoresin human evaluation in Figure 7.
Revised: Though the style is similar to COCO, the generation is not as accurate as for the non-ﬁne-tuned version, which also corresponds to the scores in human evaluation in Figure 7.

Original: 5.3 Human Evaluation
Revised: 5.3 Human Evaluation

Original: As the most persuasive metric, we conduct a large-scale human evaluation following the settingin CogView [3] (See Appendix for details).
Revised: As the most persuasive metric, we conduct a large-scale human evaluation following the setting in CogView [3] (See Appendix for details).

Original: The experiments include a total of 4,600 groups ofcomparison on COCO captions between some public available text-to-image works, including DFGAN [27], LAFITE [37], CogView [3], CogView2 (including its ﬁnetuned version on COCO) andthe recovered ground truth after VQVAE.
Revised: The experiments include a total of 4,600 groups of comparison on COCO captions between some public available text-to-image works, including DFGAN [28], LAFITE [39], CogView [3], CogView2 (including its ﬁnetuned version on COCO) and the recovered ground truth after VQVAE.

Original: Note that the VQVAE in CogView2 is much better thanthat in CogView, which makes the recovered ground truth a stronger upper bound.
Revised: Note that the VQVAE in CogView2 is much better than that in CogView, which makes the recovered ground truth a stronger upper bound.

Original: The results aredemonstrated in Figure 7.
Revised: The results are demonstrated in Figure 7.

Original: An intriguing ﬁnding is that the ﬁnetuned CogView2, although with muchbetter FID, performs worse than the original model.
Revised: An intriguing ﬁnding is that the ﬁnetuned CogView2, although with much better FID, performs worse than the original model.

Original: We guess that the model might ﬁt the style ofcomplex scenes in COCO, but the generated samples with isolated subjects could be preferred by theannotators.
Revised: We guess that the model might ﬁt the style of complex scenes in COCO, but the generated samples with isolated subjects could be preferred by the annotators.

Original: Discussion
Revised: 5.4 Analysis of the Speed and FLOPs of LoPAR

Revised: As discussed in § 1, our motivation is to increase the degree of parallelism for inference acceleration, even with more FLOPs. Autoregressive generation with cached hidden states have the same FLOPs with a teacher-forcing forward step, but is much slower (858ms vs 225.9s in CogView2 scale). For

Revised: LoPAR, it is exactly N ( N = 6 in our setting) times and FLOPs of forward steps.

Revised: We compare the inference speed of super-resolution stage with different strategies in Table 2.

Original: [Figure]
Revised: [Table]

Revised:  Discussion

Original: Autoregressive or Diffusion?
Revised: Autoregressive or Diffusion?

Original: Although GPTs achieved great success in text generation, diffusionmodels are becoming increasingly popular in image generation.
Revised: Although GPTs achieved great success in text generation, diffusion models are becoming increasingly popular in image generation.

Original: Here we compare diffusion modelswith autoregressive models from the aspect of speed, the largest disadvantage of the autoregressivemodels discussed in the section 1.
Revised: Here we compare diffusion models with autoregressive models from the aspect of speed, the largest disadvantage of the autoregressive models discussed in the section 1.

Original: With the same architecture, diffusion models require more FLOPsbut have a high degree of parallelism.
Revised: With the same architecture, diffusion models require more FLOPs but have a high degree of parallelism.

Original: They can also make a trade-off between the quality and timeconsumption by manually scheduling the stride of sampling.
Revised: They can also make a trade-off between the quality and time consumption by manually scheduling the stride of sampling.

Original: For example, Glide [18] samples 250diffusion steps for evaluation, and 27 steps for interactive sampling, to reduce the latency to 15s.
Revised: For example, Glide [19] samplesdiffusion steps for evaluation, and 27 steps for interactive sampling, to reduce the latency to 15s.

Original: Autoregressive models must generate the image token-by-token, but our LoPAR can upsample theimage with a high parallelism degree, so that (potentially) we can reduce the time cost by introducingmore hierarchies to design models much faster than diffusion models.
Revised: Autoregressive models must generate the image token-by-token, but our LoPAR can upsample the image with a high parallelism degree, so that (potentially) we can reduce the time cost by introducing more hierarchies to design models much faster than diffusion models.

Original: Comparison between DALL-E-2 and CogView2.
Revised: Comparison between DALL-E-2 and CogView2.

Original: DALL-E-

Revised: DALL-E-

Original: [26] is a recently released concurrentwork for text-to-image generation on 1024 × 1024 resolution.
Revised: [27] is a recently released concurrent work for text-to-image generation on 1024 × 1024 resolution.

Original: Although its probabilistic model andarchitecture are quite different from those of CogView2, they share the same spirit – hierarchicalgeneration.
Revised: Its probabilistic model and architecture are quite different from those in CogView2. But both models share the same spirit – hierarchical generation.

Original:  CogView2 is able to synthesize similar scenes according to the limited demos of DALL-E2, e.g. “lion teacher” (Figure 1) vs. “panda scientist” (DALL-E-2), even though CogView-2 is trainedonly about 5% of the total data used by DALL-E-2.Its main difference compared to CogView2 is athird-level super-resolution and a “zeroth”-level image prior generation.
Revised: The difference is that DALL-E-2 adopts an additional third-level super-resolution and a generation prior, which help result in potential quality gain, but also lead to expensive resourceconsuming. CogView2 is able to synthesize similar scenes according to the limited demos of DALL-E-2, e.g. “lion teacher” (Figure 1) vs. “panda scientist” (DALL-E-2), considering CogView2 is trained using only 5% of the total data (650M text-image pairs) by DALL-E-2.

Original: Since to train a third-levelsuper-resolution is resource-consuming and slightly engineering, we leave it for future work.
Revised: For future, CogView2 can also adopt the third-level super-resolution and the prior, though it is engineering mostly.

Original:  Conclusion
Revised:  Conclusion

Original: The breakthrough in the text-to-image domain is made by autoregressive models.
Revised: The breakthrough in the text-to-image domain is made by autoregressive models.

Original: However, the slowgeneration and high complexity hinder researchers attempts to improve the quality in this direction.
Revised: However, the slow generation and high complexity hinder researchers attempts to improve the quality in this direction.

Original: In this paper, we put forward an approach based on hierarchical transformers to help autoregressivemodels remedy these disadvantages, and bridge the gap between text-to-image pretraining and recentvisual representation learning methods.
Revised: In this paper, we put forward an approach based on hierarchical transformers to help autoregressive models remedy these disadvantages, and bridge the gap between text-to-image pretraining and recent visual representation learning methods.

Original: Broader Impact.
Revised: Broader Impact.

Original: The advancement of text-to-image generation, especially text-guided image editing,will ease the creative efforts of artists and designers, while also causing a risk of misinformation,leading to permanent damages to the reliability of web photos.
Revised: The advancement of text-to-image generation, especially text-guided image editing, will ease the creative efforts of artists and designers, while also causing a risk of misinformation, leading to permanent damages to the reliability of web photos.

Original: However, it is possible to train aclassiﬁer to distinguish the real and CogView2-generated images according to the texture features.
Revised: However, it is possible to train a classiﬁer to distinguish the real and CogView2-generated images according to the texture features.

Original: Checklist
Revised: Acknowledgments and Disclosure of Funding

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and

Original: scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See section
Revised: We would like to thank Zhao Xue and Sha Yuan for the help on collecting the dataset, Hanxiao Qu for maintaining the machines, and Yue Cao and Chang Zhou for their useful discussion, Zhendong

Original: (c) Did you discuss any potential negative societal impacts of your work? [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: Zhang for releasing an initial version of CUDA local attention. Funding in direct support of this work: GPU hours donated by BAAI, NSFC for Distinguished Young

Original: them? [Yes]
Revised: Scholar (61825602).

Original: If you are including theoretical results...
Revised: A High-resolution Text-guided Inﬁlling

Original: (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]
Revised: Although CogLM in nature supports text-guided inﬁlling as illustrated in Figure 3, we ﬁnd that for small patches, the model tends to only consider the coherence of context and ignore the text, because the granularity of the CogLM is only 20 × 20 .

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: Our solution is to magnify the region to an appropriate size and then run the inﬁlling and super-resolution.

Original: mental results (either in the supplemental material or as a URL)?
Revised: Specially, the process can be divided into the following steps:

Original: [No]
Revised: Calculate the square bounding box of the given mask region. Scale the size of the bounding box by 1.4 (as a default hyperparameter). If the box already covers the whole image, run the normal completion by CogLM and super-resolution; else magnify the patch in the boxed region to 480 × 480 , run the normal completion by CogLM and super-resolution only for this region, and scale the result back to the original size. Replace the masked region to the generated results.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: We show a few examples of text-guided inﬁlling in Figure 8. The examples show the potential of CogView2 on the applications like portrait editing. B Details of the Tokenizers

Original: how they
Revised: Here we introduce more details about our uniﬁed tokenizer, icetk. B.1 Text Tokenizer The text tokenizer is trained based on the sentencepiece 3 (unigram algorithm [16]) on a mixed corpus of both English and Chinese. The corpus consists of 25GB plain text – half English, half Chinese.

Original: were chosen)?
Revised: We divide the extracted tokens into four categories: common, English, Chinese, and special, by carefully selection according their unicode encoding, and assign them with consecutive ID numbers. The dictionary size of icetk is 150,000. The ﬁrst 20,000 tokens are image tokens (discussed below). There are 100 common tokens, including punctuations, numbers and self-deﬁned tokens, e.g. <unk> .

Original: [Yes]
Revised: The No. 20,100 to No. 83,822 tokens are English tokens and the No. 83,823 to No. 145,653 tokens are Chinese tokens.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: The rest are special tokens, e.g. α . We can disable the generation of part of tokens via a sampling mask, e.g. only generating English tokens for image captioning.

Original: ments multiple times)?
Revised: can be seen as

Original: [No]
Revised: We can ﬁnd that improve the images with more details, but sometimes not very accurate, e.g. the mouth in the example. B.2 Image Tokenizer The image tokenizer is a multi-compression-rate VQVAE 4 .

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: The main differences between the image tokenizer of CogView2 and that of CogView are mainly the perceptual loss and the multi-compressionrate design.

Original: [No]
Revised: The perceptual loss [37] is proposed to measure the matching degree of the human perception and the texture of a given image.

Original: If you are using existing assets (e
Revised: VQGAN [7] uses it as one of the loss term for GAN training. We surprisingly ﬁnd that the perceptual loss might account for the majority of the texture improvement of VQGAN over the VQVAE. However, VQGAN, although proved to achieve better

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: FID, sometimes produces bad cases on important elements of the images, e.g. the face. Figureshows an example for comparison between the methods.

Original: [Yes] (b) Did you mention the license of theassets? [No]
Revised: Therefore, we ﬁnally determine to use the

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: L 1 , MS-SSIM [32] and perceptual loss to train a VQVAE as the image tokenizer as follows,

Original: [N/A]
Revised: [Equation]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: where ˆ x is the recovered output after the VQVAE. The multi-compression-rate design means that our image tokenizer can compress the image with different compression rates, speciﬁcally 4 2 × , 8 2 × or 16 2 × .

Original: using/curating? [No]
Revised: Although all of our experiments in

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: CogView2 only use the 8 2 × compression rate, i.e., 160 2 → 20 2 and 480 2 → 60 2 .

Original: [No] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: We implement it by training three image tokenizer with a shared dictionary and low-level-layer parameters, which is illustrated in Figure 10. The architecture basically follows the released VQGAN [7] with self-attention layers.

Original: [Yes]
Revised: We train the model on ImageNet to convergence.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: We follow the setting of CogView [3] to conduct human evaluation.

Original: Board (IRB) approvals, if applicable?
Revised: 50 captions from MS COCO are randomly selected.

Original: [N/A]
Revised: The evaluators need to score each image (from 1 to 5) from three aspects: the image clarity, the texture quality and the relevance to the caption, and give an overall score from 1 to 10. The evaluators are also asked to select the best one from the images.

Original: (c) Did you include the estimated hourly wage paid to participants and  the total amount spent on participant compensation?
Revised: We recruited 100 evaluators, and 92 of them ﬁnally ﬁnished the tasks. Each evaluator will scoreimages and get paid with 75 yuan.

Original: [Yes]
Revised: The screenshot for the evaluation website are shown in Figure 11. We also ﬁnd a very small part of participants did not carefully score the images, but randomly select it to quickly ﬁnish the task. This may account for that even DF-GAN wins in 1.26% questions.