Locally Hierarchical Auto-Regressive Modeling for Image Generation
Abstract
We propose a locally hierarchical auto-regressive model with multiple resolutions of discrete codes.
In the ﬁrst stage of our algorithm, we represent an image with a pyramid of codes using Hierarchically Quantized Variational AutoEncoder (HQVAE), which disentangles the information contained in the multi-level codes.
For an example of two-level codes, we create two separate pathways to carry high-level coarse structures of input images using top codes while compensating for missing ﬁne details by constructing a residual connection for bottom codes.
An appropriate selection of resizing operations for code embedding maps enables top codes to capture maximal information within images and the ﬁrst stage algorithm achieves better performance on both vector quantization and image generation.
The second stage adopts Hierarchically Quantized Transformer (HQ-Transformer) to process a sequence of local pyramids, which consist of a single top code and its corresponding bottom codes.
Contrary to other hierarchical models, we sample bottom codes in parallel by exploiting the conditional independence assumption on the bottom codes.
This assumption is naturally harvested from our ﬁrst-stage model, HQ-VAE, where the bottom code learns to describe local details.
On class-conditional and text-conditional generation benchmarks, our model shows competitive performance to previous AR models in terms of ﬁdelity of generated images while enjoying lighter computational budgets.
Introduction
Deep generative models [1, 2, 3] have become the fundamental basis for synthesizing diverse images of high-resolution from various forms of input supervision, including class conditions [2, 4], text prompts [5, 6] and segmentation maps [7] to name a few.
Most approaches in this paradigm are roughly categorized into two groups depending on whether they explicitly maximize data likelihood or not.
GANs implicitly estimate underlying data distribution to generate novel and crispy samples, but it typically suffers from a lack of diversity [1].
In contrast, it is widely accepted that likelihood-based models are capable of generating diverse examples, compared to GANs, since they aim to cover all modes of a data distribution [8].
This work focuses on auto-regressive models among many likelihood-based approaches motivated by the existing works validating scalability [5, 9].
Auto-Regressive (AR) models factorize a data likelihood into a product of conditionals in the predeﬁned generation order.
Since AR models already have mode-coverage property, we only need
[Figure]
to make minimal changes in architectures and optimization methods for the applications to various domains including audio, language, and images.
However, regardless of the modality, training AR models in a high-dimensional space is computationally expensive since each conditional needs to consider many past observations.
In the context of image generation, VQ-VAE [10] introduces a two-stage learning process, where the ﬁrst stage represents an image by a discrete code map while the second stage constructs AR priors [2, 11] on discrete sequences.
To make this two-stage framework effective, a sequence of the discrete codes has to hold sufﬁcient information for highquality reconstruction of an original image while the sequence length should be reasonably short for efﬁcient training of AR models.
In this study, we postulate that leveraging the multi-scale structure is a simple but effective way for representing an image as discrete codes with AR modeling.
We aim to disentangle the information residing in the hierarchy and design an efﬁcient model to generate unseen images by exploiting the multi-level structure maximally.
To achieve this goal, we propose locally hierarchical auto-regressive modeling by extending two-stage AR modeling.
We ﬁrst extend VQ-VAE to produce a two-level pyramid of discrete codes, where the top code contains high-level coarse information and the bottom encodes the low-level ﬁne details.
Using these locally hierarchical discrete representations, our efﬁcient AR model generates a novel example.
Speciﬁcally, the top codes are sequentially generated in a predeﬁned order, and multiple bottom codes associated with a top code are decoded in parallel.
The main contributions of the proposed model are summarized below:
• We propose HQ-VAE (Hierarchically Quantized Variational Auto-Encoder), which encodes an image using a pyramid of two-level codes through hierarchical vector quantization.
To improve the utility of the two-level codes, we disentangle the information contained in the codes, where top codes capture high-level structural information in images while bottom codes carry the remaining low-level details.
• We introduce HQ-Transformer (Hierarchically Quantized Transformer), which is an AR model to generate images based on hierarchical discrete codes.
We predict a sequence of locally hierarchical codes efﬁciently by maximizing its likelihood based on the conditional independence assumption imposed on bottom codes.
• The proposed algorithm achieves competitive performance on class- and text-conditional image generation tasks while yielding high throughput in sample generation, compared to various types of generative models.
Figure 1 illustrates several examples of generated images by the proposed algorithm.
We also present the synthesized images based only on top and bottom codes together, which clearly exhibits the proper disentanglement of image contents and low-level details.
Related Works
This section describes existing approaches closely related to our algorithm and highlights the differences between the prior works and ours.
Discrete representations for autoregressive models Learning AR priors based on discrete representations turn out to be effective in language modeling [9, 12].
VQ-VAE [10] pioneers the two-stage image generation process, where the ﬁrst stage represents an image by a set of discrete codes and the next stage constructs AR priors [2, 11] on the discrete sequences.
Following VQ-VAE, many approaches aim to further reduce the sequence length and improve the architecture [13, 14] or training objectives [7] of the ﬁrst stage.
Speciﬁcally, VQ-GAN shortens the sequence length by using the perceptual and adversarial losses and encoding the semantic information in the codes effectively.
However, these approaches still exhibit the fundamental trade-off between code sequence length and generated image quality due to a lack of expressiveness of the single-scale vector quantization with the reduced length of the code sequence.
Hierarchical representations for autoregressive models
The construction of hierarchical codes is an effective way to control the fundamental trade-off related to the length of the code sequence.
There exist several image representation and generation methods based on hierarchical discrete codes [8, 15, 16].
VQ-VAE-
[8] and HAM [15] make use of the hierarchical relationship between the codes for unseen image generations in a coarse-to-ﬁne manner.
In particular, to tackle the tradeoff issue, VQ-VAE-2 [8] allocates a heavier model for processing coarse top codes and a lighter model for dealing with ﬁne bottom codes.
RQ-Transformer [17] encodes an image into a stack of code maps by iteratively quantizing residual vectors between latent features and code embeddings, effectively reducing the spatial resolution of the code maps without degrading reconstruction quality.
Hierarchical VQ-VAE [16] encodes structural and textual information of an image using different types of codes by training two separate VQ-VAEs, where one learns with the original images and the other employs the residuals between the original and reconstructed images.
The main idea of
Hierarchical VQ-VAE is about how to disentangle the information across the code levels but not about how to train AR models based on this set of codes.
Our approach constructs a pyramid of discrete codes and exploits the codes using AR models to generate high-ﬁdelity images.
Diffusion models Diffusion models [3, 18, 19] iteratively denoise a sample from white noise by seeing the contexts in a bidirectional manner.
Most diffusion models have been implemented based on the U-Net [20] style architecture, meaning that the multi-scale feature of an image has been naturally incorporated.
However, they suffer from low throughput in the generation process because, even though the inference requires a lot of iterative steps, the intermediate computation results between the denoising steps are not reusable, unlike AR models.
Locally Hierarchical Auto-Regressive Models
This section describes the details of the proposed method for image generation.
Our approach exploits two levels of codes—top and bottom—for the representation of an image contrary to existing twostage approaches based on AR models.
The proposed architecture encourages the top code to model the overall structure of an image while facilitating the bottom code to complement the remaining details of the image.
To this end, we propose Hierarchically Quantized Variational AutoEncoder (HQ-VAE) for the image vector quantization in the ﬁrst stage followed by HQ-Transformer to learn the distribution over a sequence of the top and bottom codes in the second stage.
HQ-Transformer achieves better perceptual quality and sampling speed using the underlying hierarchical structure.
Figure 2 illustrates the two-stage procedure of our algorithm based on an AR model.
3.1 Stage 1: Hierarchically Quantized VAE (HQ-VAE)
We present the architecture and the optimization method of the proposed HQ-VAE, for which we ﬁrst review the vector quantization technique and then discuss our hierarchical quantization module.
3.1.1 Notations for vector quantization
Vector quantization [21] maps a d -dimensional continuous feature vector to a discrete code, which is given by identifying the nearest code embedding of the feature vector in a codebook.
The codebook
E n c o d e r
STAGE 1 (HQ-VAE)
Downsample
Quantizer
Upsample
QuantizerD
e c o d e r e top e bot
Image (256x256)
Reconstruction
STAGE 2 (HQ-Transformer) sos
Main Transformer uuu n h 2 h 1 hh n
Input Embedding Transformer t n b (n,1) b (n,2) b (n,3) b (n,4)
Prediction Head Transformer t n+1 t n+1 t n+1 t n+ b (n+1,1) b (n+1,2) b (n+1,3) b (n+1,4) t n+
[Figure]
Causal Attention Blocks
Bidirectional Attention Blocks
Mixed Attention Blocks t 1 tt 3 tb (1,1) b (1,2) b (1,3) b (1,4)
b (2,1) b (2,2) b (
2,3) b (2,4) b (3,1) b (3,2) b (3,3) b (3,4)
[Figure]
[Figure]
is a collection of tuples C =
{ ( s, e s ) } , where e s ∈ R d e denotes the embedding of code s and n C is the size of the codebook C .
Supposing that z ij is a d -dimensional vector at a spatial location ( i, j ) in a feature map z ∈ R l × l × d , a code s ij is given by
[Equation]
where C [ · ] returns the code embedding of an input code and Q ( · ; C ) denotes the quantization process based on a codebook C .
Note that, since every element in a feature map has its own code, the spatial resolution of the code map, denoted by s , is the same as that of the feature map.
3.1.2 Hierarchical vector quantization
Our approach adopts a hierarchical vector quantization scheme to encode input data using two levels of discrete codes—top and bottom denoted by t and b , respectively.
Ideally, the combination of the two-level codes is expected to maintain the full representation power of an original feature map.
In addition, we desire the two kinds of codes to have sufﬁciently disentangled and heterogeneous information; for example, top codes hold the high-level concept of inputs while bottom ones represent remaining low-level details.
To these ends, our network creates two separate pathways designated for the top and bottom codes and makes the branch for bottom codes constrained to have residual information.
In speciﬁc, our hierarchical vector quantization transforms a feature map z ∈ R rl × rl × d into a tuple of two code maps ( t , b ) , where t ∈ Z l × l and b ∈
Z rl × rl with an integer scaling factor r ∈ { 1 , 2 , . . . } .
Our scheme ﬁrst captures the high-level information of a feature map by quantizing its downsampled version using the top codes and computing their embeddings as follows:
[Equation]
where C top denotes the codebook of top codes and Downsample ( · ) is a generic pooling operation to reduce the spatial resolution of a feature map.
Given the top code map t , the bottom codes and its embeddings are derived by
[Equation]
where C bot is the codebook of bottom codes and Upsample ( · ) is any upsampling operation to increase the spatial resolution of a feature map.
Figure 2(left) illustrates this hierarchical quantization procedure.
Note that the residual vector between the original feature map and the upsampled top code embedding map, denoted by z bot , involves sampling and quantization errors.
The bottom codes with their embeddings should compensate for the errors and improve the reconstruction quality of the original feature map when combined with the top-level information.
o r
i g i na l
Avg.
Pool
Quantizer
Nearest Interp.
Quantizer
Encoder
Decoder 16 x 16 x K 8 x 8 x K 16 x 16 x K - (a) Average pooling and nearest interpolation o r i g i na l
PixelUnShuffle
Quantizer
PixelShuffle
Quantizer
Encoder
Decoder16 x 16 x K 8 x 8 x 4K 16 x 16 x K (b) Pixel-unshufﬂe and pixel
[Figure]
shufﬂe
One question is which operations are appropriate for Downsample ( · ) and Upsample ( · ) .
We observe that non-learnable layers give favorable solutions for the resizing operations compared to naïve learnable counterparts.
Straightforward choices for those operations are the average pooling and the interpolation with nearest neighbors.
Despite the reasonable performance given by this strategy,
Figure 3a exhibits that the overall structure reconstructed by the top code embeddings is below expectation.
Instead, it turns out that a combination of pixel-unshufﬂe and pixel-shufﬂe operations [22] effectively pushes more information into the top codes while making the bottom codes focus on local details.
Given the feature map z ∈ R rl × rl × d and a scaling factor r , the pixel-unshufﬂe operation returns a tensor in the shape of l × l × r 2 d , encoding high-level information of the original feature map.
To compute the residual, we pixel-shufﬂe the top code embedding and restore the original dimensionality of z .
Figure 3 clearly demonstrates that the top codes given by the pixel-unshufﬂe followed by the quantization hold the primary information of an input image more than the ones given by the average pooling.
3.1.3 HQ-VAE: architecture and optimization
Overall structure HQ-VAE consists of three parts: encoder E ( · ) , decoder D ( · ) , and hierarchical quantizer H ( · ; r ) , as shown in Figure 2(left), where r is the additional scaling factor for the top code.
Given an image x , an encoder E ( · ) extracts the feature map z .
Then, the hierarchical quantizer
H ( · ; r ) is applied to the feature map, providing the top and bottom codes ( t , b ) and their associated code embeddings: e top and e bot .
Now, the decoder D ( · ) takes the top and bottom code embeddings to reconstruct the input image.
Objective function To train three parts in HQ-VAE, our objective function is composed of the reconstruction loss introduced in VQ-VAE [10] as well as adversarial and perceptual losses to improve the ﬁdelity of reconstructed images [7].
Here, we only present the reconstruction loss as below:
[Equation]
where sg ( · ) refers to a stop-gradient operation, and β is set to 0.25 as suggested in [10].
The ﬁrst term measures (cid:96) 2 distance between the reconstructed and original images.
The second term enforces both the top and bottom code embedding to be located close to the inputs of the quantizers.
To update the codebook, we make use of exponential moving average (EMA) to accelerate convergence as discussed in [10].
Implementation details We adopt the same network structures of the encoder and the decoder as
VQ-GAN [7] except two minor modiﬁcations.
To increase throughput in training, we set the stride of the ﬁrst convolution layer in the encoder and the last transposed convolution layer in the decoder as 2, instead of 1 in the original implementation.
We replace the batch normalization [23] in the discriminator by the group normalization [24] to achieve consistent results regardless of batch sizes.
3.2 Stage 2: Hierarchically Quantized Transformer (HQ-Transformer)
This section presents HQ-Transformer (see Figure 2(right)) to autoregressively model the top and bottom codes by fully exploiting their spatial relationships obtained from the ﬁrst stage.
3.2.1 Formulation
Our objective is to maximize the log-likelihood of a tuple of t and b .
Given the scaling factor r for the top, the resizing operation aligns a single top code t i with r 2 bottom codes b ( i,k ) , where 1 ≤ k ≤ r 2 .
Figure 2 illustrates an example of top and bottom codes with r = 2 (in the input embedding transformer part), where all bottom codes aligned with a single top code t i are b i =
( b ( i, 1) , b ( i,r 2 ) ) .
The likelihood of the top and bottom codes is given by
[Equation]
where the inequality symbol ( < i ) denotes a collection of all indices less than i .
The conditional is further factorized into
[Equation]
The last term in (6) is maximized by predicting the current position given the previous top and bottom codes.
By assuming that the bottom codes are conditionally independent given the associated top code, the ﬁrst term of the right-hand side is formulated more efﬁciently as follows:
[Equation]
Note that the assumption we made here is reasonable because the bottom codes are supposed to contain local details while the top codes are enforced by our stage 1 model, HQ-VAE, to encode a global landscape.
3.2.2 Model
Similar to GPT-like architectures [9, 25], HQ-Transformer consists of three sub-modules based on the transformer architecture, which are referred to as Input Embedding Transformer (IET), Main
Transformer (MT), and Prediction Head Transformer (PHT), as illustrated in Figure 2(right).
IET generates an embedding to be used as an input for MT, and the hidden state given by MT is fed into
PHT to generate top and bottom codes, which are given back to IET in the next step.
We discuss each of the three modules in the rest of this section.
Input Embedding Transformer (IET) IET transforms a tuple of ( t i , b i ) into a single embedding vector u i , ∀ i .
To implement this, we ﬁrst construct a local embedding by summing up the code embedding and the positional encoding, which is given by
[Equation]
where e top,in i and e bot,in ( i,k ) are the learnable code embedding of t i and
b ( i,k ) , respectively, and PE top [ · ] and PE bot [ · ] denote the learnable positional encoding for the top and bottom 2 .
Note that the code embeddings are learned within this IET module, by which we construct the codebooks C top,in and
C bot,in .
These local embeddings are reﬁned by a bidirectional transformer,
[Equation]
where the structure is the same as GPT-2 [9] and we employ a single GPT-2 block composed of a selfattention followed by an MLP block.
Then, the ﬁnal input embedding u i for the main Transformer block is simply given by averaging all updated local embeddings as
[Equation]
Main Transformer (MT)
This module updates the outputs of IET by a stack of N MT causal self-attention and MLP blocks:
[Equation]
where the sequence length is equal to the number of top codes and the start-of-sentence token (SOS) is employed as the ﬁrst input of the MT ( · ) function.
This function also has the same structure as
GPT-2.
Prediction Head Transformer (PHT)
This module predicts the top and bottom codes from the hidden representation derived by the main transformer, where the bottom codes are generated in parallel based on their associated top code and a hidden representation from MT.
In speciﬁc, we ﬁrst deﬁne the input sequence of PHT as below:
[Equation] [Equation]
where e top,head i denotes the learnable code embedding of t i deﬁned within PHT and PE head [ · ] denotes the learnable positional encoding for top and associated bottom codes deﬁned within PHT.
This sequence is then updated to produce the conditional probabilities of the top and bottom codes as follows:
[Equation]
where each output probability is deﬁned as
[Equation]
Note that we predict the bottom code distributions in parallel.
In PHT, we employ a self-attention block of a mixture of the positions of a top code and multiple bottom codes, which is conditioned on the top position while the attention between bottom positions is unconditional.
Compared to the full autoregressive modeling on the bottoms, such a parallel decoding scheme for bottom codes leads to a speed-up in sampling and even better image generation quality.
We will discuss its beneﬁts in the experiments section.
3.2.3 Computational complexity of HQ-Transformer
Given the top code t ∈ Z l × l and the bottom code b ∈
Z rl × rl , where r is a scaling factor for the quantization of top codes, the length of the top codes is L top =
l 2 and the number of codes in a local patch is L local = 1 + r 2 because there exist r 2 bottom codes associated with a single top code.
The time complexities of IET, MT, and PHT are then given by O ( N IET L top L 2 local ) , O ( N MT L 2 top ) and
O (
N PHT L top L 2 local ) , respectively, where N IET , N MT , and N PHT are the numbers of attention blocks in individual modules.
The time complexity of MT dominates those of the other two modules because
N MT is larger than N IET and N PHT .
Because the
[Table] [Table]
3.3 Sampling from HQ-Transformer and HQ-VAE
For inference
[Table]
we ﬁrst generate discrete codes using HQ-Transformer and synthesize new images using HQ-VAE based on the codes.
We begin with the start-of-sentence (SOS) token and obtain its hidden representation from MT in HQ-Transformer
[Table]
where the class-conditional SOS is employed in the case of class-conditional generation.
Then, PHT takes the hidden representation as its input embedding at the ﬁrst step and predicts a top code distribution.
We sample a top code from the distribution and forward it to the next step of PHT to obtain the bottom codes, which are actually acquired in parallel by forwarding the top code embedding to the multiple next steps.
The top and bottom codes are given to IET, which outputs the embedding for the input to MT.
We repeat this procedure in an auto-regressive manner until we obtain all the top and bottom codes.
Once completing the generation of the codes, we ﬁnally produce a synthetic image using the decoder of HQ-VAE based on the discrete codes.
Experiments
We evaluate our models, referred to as HQ-TVAE hereafter, on class- and text-conditional image generation tasks.
4.1 Conditional Image Generation
Experiment settings We train our models on 1.2M images in the train split of ImageNet [29] for class-conditional image generation.
For text-conditional tasks, we employ 15M image-text pairs in
Conceptual Caption (CC) [30] and Conceptual-12M [31].
In the training phase of both cases, we resize the short axis of an image to 256 and randomly crop it to acquire a 256 × 256 image.
In testing, we also resize the short axis to 256 and crop the image at the center.
The scaling factor in HQ-VAE is set to 2, i.e. , r = 2 , to produce the top and bottom codes of 8 ×and 16 × 16 resolution by default, respectively.
We test three versions of HQ-Transformer by varying hyperparameters: (a) N MT
=
12 , and N PHT
= 4 for the smallest model, (b) N MT
= 24 and N PHT =for the mid-size model, and (c) N MT = 42 , and N PHT = 6 for our largest model.
In all cases,
N IET = 1 and other parameters related to the network size are ﬁxed.
We describe more details about the network architecture in the supplementary document.
As evaluation metrics, we adopt Fréchet
Inception Distance (FID) [32] and precision & recall (PR) [33] for class-conditional generation, and
[Figure]
[Table]
FID and CLIP-score [34] for the text-conditional generation task.
We measure the throughput of sample generation on a single Tesla A100 GPU
[Table]
Class-conditional image generation Table 1 presents that our largest model, HQ-TVAE (L), achieves the best FID (7.15) compared to other two-stage AR models with similar sequence lengths such as VQ-GAN ( 16 × 16 ) and RQ-Transformer ( 8 × 8 × 4 ).
We observe that HQ-TVAE reports the highest throughput in sample generation than other models because it generates r 2 bottom codes in parallel given an associated top code.
Compared to ADM [26], our smallest model (S) generates images of similar quality while achieving 650 times throughput.
We also conduct rejection sampling in all the models based on ResNet-101 [35] with an acceptance rate of 0.25 and report the results within the parentheses in Table 1.
Figure 4 demonstrates the qualitative results of our approach, which also include the generated images based only on top codes to visualize the representation disentanglement.
Interestingly.
contrary to typical generative models, HQ-TVAE restricts the diversity of generated images within details by sampling only bottom codes from HQ-Transformer with ﬁxed top codes.
Text-conditional image generation Table 2 presents FIDs and CLIP-scores of several models measured on the CC3M validation split, and the throughput in sample generation.
The results show that our model is comparable to RQ-Transformer in terms of FID (12.86 vs. 12.33) and CLIP-score (0.26 vs. 0.26) while achieving 18%p higher throughput than RQ-Transformer.
4.2 Analysis of HQ-VAE and HQ-Transformer Models
Comparison with other stage 1 methods Table 3 compares HQ-VAE implementations with two different feature map sizes (shown in the parentheses) from the encoder with the existing stagemodels, where we measure FIDs between the original and reconstructed images, referred to as rFID.
The last column of the table reports the utilization of codebook(s) for the representations of the images in the ImageNet validation dataset, and our algorithm turns out to make full use of the codebooks.
rFID from HQ-VAE ( 8 × 8 + 16 × 16 ) is better than the baselines with the similar sequence length, e.g. , RQ-VAE ( 8 × 8 × 4 ) and RQ-VAE ( 8 × 8 × 5 ).
In addition, HQ-VAE ( 16 × 16 + 32 × 32 )
[Table] [Table]
[Table]
outperforms ViT-VQGAN ( 32 × 32 ) although we admit that ViT-VQGAN can be implemented more efﬁciently based on the hierarchical code maps.
Ablation study on HQ-VAE Table 4 supports the rationale of our choice, showing that resizing with pixel-unshufﬂe and -shufﬂe outperforms the average pooling followed by the nearest interpolation by 0.53 in FID since it prevents information loss during the downsampling operation.
We also observe that longer training boosts rFID performance from 3.14 to 2.61.
Note that learnable resizing turns out to be not effective, where we adopt a simple network with single convolution and transposed convolution layers with kernel size 2 and stride 2, for downsampling and upsampling, respectively.
We impose the same loss function on the reconstructed images using only the top codes to avoid top code collapse.
Refer to the supplementary document for a more detailed analysis.
Ablation study on HQ-Transformer Table 5 compares the effect of decoding policies and highlights the beneﬁt of the proposed locally hierarchical decoding.
VQ-VAE-
[8] ﬁrst generates the entire top codes and then derives the bottom codes one by one with global conditioning on the sequence of the top codes.
RQ-Transformer [17] sequentially generates a top code and the relevant bottom codes, locally conditioned on the generated codes from the previous steps.
We observe that our decoding strategy achieves better FID than the sequential methods since error accumulation during sampling is alleviated by our parallel decoding of bottom codes.
We also validate the effects of soft-labeling as an alternative to one-hot labeling in the cross-entropy loss function for training
HQ-Transformer as proposed in RQ-Transformer [17].
The use of soft labels with temperature τ =in our case improves performance as similarly observed in [17].
Multi-level extensions Table 6 present that three-level architectures outperform the two-level counterparts.
For these experiments, we train two different combinations of three-level codes { (4 × 4) , (8 × 8) , (16 × 16) } and { (8 × 8) , (16 × 16) , (32 × 32) } .
The results show that the use of three-level codes is helpful for improving performance at the early stage of training (15 epochs for training HQ-VAE).
However, they fail to achieve additional gains with more iterations probably due to the lower usage of top codes as shown in the “code utilization" column of the table.
Detailed architectures for multi-level extensions are described in the supplementary document.
Conclusion
We presented a two-stage auto-regressive model, which consists of HQ-VAE and HQ-Transformer, to generate images based on hierarchically quantized codes.
HQ-VAE represents an image by two-level discrete codes with different spatial resolutions, where it encodes the high-level structure of an image using the top codes and the low-level information using the bottom codes.
HQ-Transformer models the multi-level codes efﬁciently and the combination of the two modules successfully generates unseen images.
Experiments conﬁrm the capability and potential of our model for high-quality and high-resolution image generation with a fraction of computational cost compared to existing methods.
Acknowledgement This work was partly supported by IITP grant [No.2022-0-00959, (Part 2)
Few-Shot Learning of Causal Inference in Vision and Language for Decision Making, 2021-001343, Artiﬁcial Intelligence Graduate School Program (Seoul National University)] and NRF grant [No.2022R1A5A708390811, Trustworthy Artiﬁcial Intelligence] funded by the Korean government (MSIT).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We discuss the limitation of our models on the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss it in the supplementary material (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state
the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We report in both the paper and the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We report in both the paper and the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The evaluation protocol for conditional generative models does not report the error bars.
Instead, we ﬁx the random seed and make experiments reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We report in both the paper and the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite and report in the supplementary material.
(b) Did you mention the license of the assets?
[Yes] We report in the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]