Locally Hierarchical Auto-Regressive Modeling for Image Generation
Abstract
Auto-Regressive (AR) modeling on image generation provides a scalable solution to synthesize high-quality images by directly optimizing the data likelihood.
For an efﬁcient AR modeling of high-resolution images, many practical approaches decompose a learning process into the following two stages: (a) representing a high-resolution image by a sequence of codes, and (b) learning an AR model on this sequence, rather than on a sequence of pixels.
This paper points out that determining the resolution of a single-scale code map triggers the trade-off between computational complexity and the ﬁdelity of generated images. For example, high-ﬁdelity image generation requires higher-resolution codes. But it needs exhaustive computation.
To mitigate this trade-off, we propose a locally hierarchical auto-regressive model with different resolutions of codes. Our model represents an image by a pyramid of codes— coarse top code for high-level structure and ﬁne bottom code for remaining details— via Hierarchical-Quantized
Variational AutoEncoder (HQ-VAE) in the ﬁrst stage.
Hierarchical-Quantized
Transformer (HQ-Transformer) in the second stage models a sequence of local hierarchies, which consist of a single top code and corresponding bottom codes. On class-conditional and text-conditional generation benchmarks, our model shows comparable performance to previous AR models in terms of the ﬁdelity of generated images with lighter computational budgets.
Introduction
Deep generative models [11, 15, 35] have become the fundamental basis for synthesizing diverse images of high-resolution from various forms of input supervision, including class conditions [21, 35], text prompts [26, 27], and segmentation maps [9] to name a few.
Most approaches in this paradigm could be roughly categorized into two types depending on whether they explicitly maximize the data likelihood or not.
GANs implicitly estimate the underlying data distribution to generate novel and crispy samples, but it typically suffers from the lack of diversity [11].
In contrast, it is widely accepted that likelihood-based models are capable of generating diverse samples, compared to GANs, since they aim to cover all modes of data distribution [28].
In this paper, we focus on auto-regressive models among many likelihood-based approaches since the scalability of auto-regressive models is validated in several works [25, 27]. [Figure]
Auto-Regressive (AR) models factorize the data likelihood into a product of conditionals in predeﬁned generation order.
Besides the mode-coverage property of AR models, they require only minimal changes to architecture and optimization for being applied to various data modalities, such as audio, natural language, and images [24, 34, 35].
However, regardless of the modality, training
AR models on high-dimensional samples is a very computationally demanding task, since each conditional needs to process many past observations.
In the context of image generation, VQVAE [36] introduces a two-stage learning process, where stage 1 represents an image by a discrete code map, and stage 2 constructs AR priors [4, 35] on discrete sequences much shorter than the image.
To make this two-stage framework effective, the codes need to have sufﬁcient information to reconstruct the original image from the discrete sequence, while the sequence length should be reasonably short to train AR models.
In this study, we postulate that leveraging the multi-scale structure of visual images is an efﬁcient way to represent an image as discrete codes for AR modeling.
Suppose that we have multi-level codes obtained from stage 1, where the high-level code contains the overall structure and the low-level one encodes the detailed information.
It now becomes possible to treat such codes differently, so an efﬁcient solution could be presented in stage 2.
This is because given the codes of low resolution, a light AR model may be enough to generate the codes of high resolution.
In order to achieve this goal, we propose locally hierarchical auto-regressive modeling extending two-stage AR modeling.
We ﬁrst extend VQ-VAE to produce a pyramid of two levels: the top and bottom codes, where the top contains the high-level information and the bottom encode the low-level details.
Using these hierarchical discrete representations, an efﬁcient AR model could be developed to generate a novel sample in a locally hierarchical manner.
Speciﬁcally, the top codes are sequentially generated in a pre-deﬁned order, and several bottom codes associated with a top code are decoded in parallel.
The main contributions of the proposed model are summarized below:
• We propose HQ-VAE (Hierarchical-Quantized Variational Auto-Encoder) which encodes an image into a pyramid of two-level codes—coarse top codes and ﬁne bottom codes— with hierarchical vector quantization.
To fully utilize two-level codes, we enforce a disentanglement where coarse top code captures the high-level structure and ﬁne bottom code to complement the remaining low-level details for the bottom code.
• We propose HQ-Transformer (Hierarchical-Quantized Transformer) built upon an assumption in local hierarchy that there exist several bottom codes conditionally independent given a top code. By predicting a sequence of local hierarchies with pre-deﬁned order, we predict ﬁne resolution codes with light computational budget.
• Our model achieves competitive performance on class- and text-conditional image generation, while observing high throughput in sample generation, compared to various types of generative models.
Related Works
In this section, we highlight the relevance and difference between previous approaches and ours.
Discrete representation for autoregressive models Learning AR priors on discrete representations have been proven to be effective in language modeling [24, 25].
In the case of continuous samples like images, we need to encode sufﬁcient information of examples in the codes, while the sequence length should be short to train AR models.
VQ-VAE pioneers a two-stage image generation process, where stage 1 represents an image by a set of discrete codes, and stage 2 constructs AR priors [4, 35] on discrete sequences.
Many approaches to reducing the sequence length have been proposed to improve the stage 1 architecture [17, 38] or training objectives [9].
VQ-GAN [9] shortens the codes by using the perceptual and adversarial losses to encode the semantic information in the codes as much as possible.
However, we observe that these approaches do not consider the multi-scale nature of an image, meaning that the receptive ﬁeld of every code has the same size. Hierarchical representation for autoregressive models To resolve the limitation of these singlescale discrete representation learning methods, many approaches to constructing hierarchical codes [10, 22, 28] have been proposed. VQ-VAE 2 [28] and HAM [10] make use of the hierarchical relationship between the codes to generate a novel image in a coarse-to-ﬁne manner. RQTransformer [18] further reduces the sequence length by converting an image into a stack of code maps.
This is constructed by iteratively quantizing the residual vector between the latent feature and code embedding, effectively reducing the latent feature map resolution without degrading the reconstruction quality.
Hierarchical VQ-VAE [22] simply encodes the structural and textual information in different types of codes by training two separate VQ-VAEs, where the ﬁrst is trained on the original images and the second on the residual between the original and reconstructed image.
This process can disentangle the information in the code levels, but how to train AR models based on this set of codes is not studied.
Constructed hierarchical codes can control the trade-off over code resolutions effectively.
VQ-VAE2 [28] assigns a heavier model for coarse top code and a lighter model for ﬁne bottom codes to trade-off the spatial resolution.
RQ-Transformer with a stack of code maps from RQ-Transformer assigns a deep transformer for dealing with spatial resolution and a shallow transformer for depth.
Our approaches also construct and exploits a pyramid of code map to effectively trade-off code resolution inspired from multi-scale nature.
Diffusion models Diffusion models [12, 15, 29] iteratively denoise a sample from the white noise by seeing the contexts in a bidirectional manner.
Most diffusion models have been implemented in a U-Net [30] style architecture, meaning that the multi-scale feature of an image is well exploited.
However, they suffer from low throughput in the generation, since the intermediate computation results between the denoising steps are not reusable, unlike AR models.
Locally Hierarchical Auto-Regressive Models
This section describes the details of the proposed architecture for image generation.
Our approach exploits two levels of codes—top and bottom codes—for the representation of an image contrary to existing two-stage approaches based on AR models.
The proposed architecture encourages the top code to model the overall structure of an image while facilitating the bottom code to complement the remaining details of the image.
To this end, we propose Hierarchical-Quantized Variational AutoEncoder (HQ-VAE) for the image vector quantization in the ﬁrst stage followed by HQ-Transformer to learn the prior over a sequence of the top and bottom codes in the second stage.
Note that HQTransformer achieves better perceptual quality and sampling speed using the underlying hierarchical structure.
Figure 2 illustrates the two stage procedure of our AR model.
3.1 Stage 1: Hierarchical-Quantized VAE (HQ-VAE)
We present the architecture and the optimization method of the proposed HQ-VAE, for which we ﬁrst review the vector quantization technique and then discuss the hierarchical quantization module.
E n c o d e r STAGE 1 (HQ-VAE)
Downsample
Quantizer
Upsample
QuantizerD
e c o d e r e top e bot
Image (256x256)
Reconstruction
STAGE 2 (HQ-Transformer) sos
Main Transformer uuu n h 2 h 1 hh n
Input Embedding Transformer t n b (n,1) b (n,2) b (n,3) b (n,4)
Prediction Head Transformer t n+1 t n+1 t n+1 t n+ b (n+1,1) b (n+1,2) b (n+1,3) b (n+1,4) t n+
[Figure]
Causal
[Figure]
Blocks
Bidirectional Attention Blocks
Mixed Attention Blocks t 1 tt 3 tb (1,1) b (1,2) b (1,3) b (1,4)
b (2,1) b (2,2) b (2,3) b (2,4)
[Figure]
(3,1) b (3,2) b (3,3) b (3,4)
[Figure]
[Figure]
[Figure]
3.1.1 Notations for Vector Quantization
Vector quantization [20] maps a d -dimensional continuous vector into a code s .
The mapping selects the nearest code embedding in a codebook, a collection of tuples { ( s, e ( s )) } s ∈ n C , where e ( s ) ∈ R d e denotes the embedding of code s and n C is the size of codebook C .
Supposing that z ij is a d -dimensional vector from the feature map z ∈ R l × l × d at the spatial location ( i, j ) , the code s ij and its embedding are given by
[Equation]
where C ( · ) returns the code embedding of an input code and s ij =
Q ( z ij ; C ) denotes the quantization process introduced for later use.
Note that, since every element in a feature map has its own code, the spatial resolution of the code map, denoted by s , is the same as that of the feature map.
3.1.2 Hierarchical Vector Quantization
The objective of hierarchical vector quantization process is to encode different levels of information using the top and bottom codes.
Intuitively, it is desirable for maximizing information in the codes for better ﬁdelity of images. To maximize the contained information in two-level of the codes, we introduce a disentanglement scheme that encourages top codes to hold high-level information and bottom codes to encode remaining low-level details.
This scheme results in maximum use of top and bottom codes. To accomplish this objective, our hierarchical quantization has two pathways for vector quantization process of top t and bottom code b .
In speciﬁc, Hierarchical vector quantization H ( · ; r ) transforms a feature map z ∈ R rl × rl × d into a tuple of two code maps ( t , b ) , where t ∈ Z l × l and b ∈ Z rl × rl with an integer-valued scaling factor r ≥ 1 .
[Equation]
where C top denotes the codebook of top codes and Downsample ( · ) is the average pooling function to reduce the spatial resolution of a feature map.
Based on t , the bottom code and its embedding are derived by
[Equation]
where C bot is the codebook of bottom codes and Upsample ( · ) is any upsampling operation to increase the spatial resolution of the feature map.
Figure 2(left) illustrates this hierarchical quantization process.
o r
i g i na l
Avg.
Pool
Quantizer
Nearest Interp.
Quantizer
Encoder
Decoder 16 x 16 x K 8 x 8 x K 16 x 16 x K - (a) Average pooling and nearest interpolation o r i g i na l
PixelUnShuffle
Quantizer
PixelShuffle
Quantizer
Encoder
Decoder16 x 16 x K 8 x 8 x 4K 16 x 16 x K (b) Pixel-unshufﬂe and pixel-shufﬂe
Our scheme ﬁrst captures the high-level information of a feature map by quantizing its downsampled version based on the top codebook.
Hence, the residual vector between the original feature map and the upsampled top code embedding map z bot contains resampling and quantization errors.
Figurecompares the reconstructed images from each of the top and bottom code embeddings, demonstrating the disentanglement enforced by the proposed hierarchical vector quantization.
We ﬁrst instantiate resampling operations in hierarchical quantization by average pooling for downsampling and nearest-neighbor interpolation for upsampling. Despite the reasonable performance given by these naive resampling operations, we observe that the overall structure reconstructed by the top code embeddings is not strong, as shown in Figure 3a. To push more information into the top codes, we have to reduce the information loss caused by the downsampling operation as much as possible. A simple yet effective solution to this challenge is to employ pixel-unshufﬂe and -shufﬂe operations [33].
Given the feature map z ∈ R rl × rl × d and a scaling factor r , the pixel-unshufﬂe operation returns a tensor with the shape l × l × r 2 d and maintains all information in the original feature map.
To compute the residual, we shufﬂe the top code embedding and restore the original dimensionality of z .
Figure 3 clearly demonstrates that the top codes constructed by the pixel-shufﬂe contain the primary information of an input image more than the ones given by pooling operations.
We observe that non-learnable layers give a straightforward solution for resampling operations over naive learnable layers.
A sophisticated design of additional loss functions for top codes is needed to avoid top code collapse because there is no incentive to encode the global information in the top code.
However, the naive design of hierarchical quantization only relies on high-resolution bottom codes and fails to exploit low-resolution top codes due to no incentive for using top codes with smaller resolutions in vector quantization.
Even if there exists an additional loss function, resampling operations with the learnable layers perform worse.
We discuss the design of the loss function on the experiment section in detail. 3.1.3 HQ-VAE: Architecture & Optimization Overall structure HQ-VAE consists of three parts: encoder E ( · ; f ) , decoder D ( · ) , and hierarchical quantizer H ( · ; r ) , as shown in Figure 2, where f is the initial downsampling factor and r is the additional scaling factor for the top code.
Given an image x ∈ R h × w × 3 , an encoder E ( · ) extracts the feature map z ∈ R hf × wf × d .
Then, the hierarchical quantizer H ( · ; r ) is applied to the feature map, providing the top and bottom codes ( t , b ) and their associated code embeddings: e top and e bot .
Now, the decoder D ( · ) takes the top and bottom code embeddings to reconstruct the input image.
Objective function To train three parts in HQ-VAE, our objective function is composed of the reconstruction loss introduced in VQ-VAE [36] as well as adversarial and perceptual losses to improve the ﬁdelity of reconstructed images [9].
Here, we only present the reconstruction loss as below:
[Equation]
where sg [ · ] refers to a stop-gradient operation, and β is set to 0.25 as suggested in [36].
The ﬁrst term measures L 2 distance between the reconstructed image and the original.
The second and third terms enforce the code embedding close to the inputs of quantizers.
Since we have two types of codes, we simply sum up the second and third terms of each type.
To update the codebook, we make use of exponential moving average (EMA), because this accelerates the convergence [36].
Implementation details We adopt the same network structures for encoder and decoder as in
VQ-GAN [9] with two minor modiﬁcations: (a) to increase throughput in training, we set the stride of the ﬁrst convolution layer in the encoder and the last transposed convolution layer in the decoder as two, instead of one in the original implementation.
(b) we replace the batch normalization [16] in a discriminator by group normalization [37] to achieve the consistent results regardless of a batch size.
3.2 Stage 2: Hierarchical-Quantized Transformer (HQ-Transformer)
In this section, we propose HQ-Transformer (see Figure 2) to autoregressively model the top and bottom codes by fully exploiting their spatial relationships obtained from the ﬁrst stage.
3.2.1 Formulation
Our objective is to maximize the log-likelihood of a tuple of t and b .
Given the scaling factor r for the top, the resampling operation aligns a single top code t i with r 2 bottom codes b ( i,k ) , where 1 ≤ k ≤ r 2 for every i . An example of top and bottom codes with r = 2 is shown in the bottom of Figure 2.
We denote all bottom codes aligned with a single top code t i as b i = ( b ( i, 1) , b ( i,r 2 ) ) .
Given that the top code map shape is l × l and the sequence order of the top is a typical raster scan order, the likelihood function can be factorized below:
[Equation]
where the notation <
i denotes a collection of j when j < i .
The conditional can be further factorized into:
[Equation]
where the last term can be maximized by predicting the current position given the previous top and bottom codes.
For the ﬁrst term, we observe that a reasonable assumption on the bottom codes leads to more efﬁcient formulation.
Here, we assume that the bottom codes are conditionally independent given the associated top code, since bottom codes contain only detailed local information if the top code has global-level information, which is enforced by our stage 1 model, HQ-VAE.
Therefore, we further expand the ﬁrst term in
Eq. 6 as
[Equation]
3.2.2 Architectural Details
Similar to GPT-like architectures [3, 25], HQ-Transformer consists of three sub-modules—input embedding, main transformer, and prediction head.
To effectively aggregate the information from the top and bottom codes for the main transformer and making a prediction, we design the input embedding and prediction head by a Transformer instead of linear projection.
Input Embedding Transformer (IET) IET transforms a tuple of ( t i , b i ) into a single embedding vector u i , for all i .
In order to do that, we ﬁrst construct a local embedding by summing up the code embedding and positional encoding:
[Equation]
where e top,in i and e bot,in ( i,k ) mean the corresponding learnable code embedding of t
i and b ( i,k ) and PE top [ · ] and PE bot [ · ] denote learnable positional encodings for the top and bottom.
Note that PE bot [0] is introduced to distinguish between the local embeddings from the top and bottom codes.
Now, these local embeddings are reﬁned by a bidirectional transformer:
[Equation]
where the structure is the same with GPT-2 [25].
And we observe that it is sufﬁcient to employ a single GPT-2 block (composed of a self-attention and MLP block).
Then, the ﬁnal input embedding u i for the main Transformer block is simply averaging all updated local embeddings:
[Equation] [Equation]
Main Transformer
This module updates the outputs of IET by a stack of N main causal self-attention and MLP blocks:
[Equation]
where the sequence length is the number of top codes and the start-of-sentence token (SOS) is located at the ﬁrst of input embedding.
Here, we use the same structure of GPT-2.
Prediction Head Transformer (PHT)
This module predicts the top and bottom codes from the hidden representation of the main transformer, where the bottom codes at the next position are generated in parallel based on their associated top code and a hidden representation from the main
Transformer.
In speciﬁc, we ﬁrst deﬁne the input sequence of PHT as below:
[Equation]
where e top,head i means the corresponding learnable code embedding of t i .
This sequence is then updated to produce the conditional probabilities of top and bottom codes:
[Equation]
where each probability is deﬁned as
[Equation]
Here, we use the self-attention mask pattern as a mixture of causal and bidirectional mask—the causal attention from the top to bottom codes and the bidirectional attention between the bottoms.
Compared to full-autoregressively modeling on the bottoms, this parallel decoding achieves faster sampling and even better generation quality.
We will discuss the effect of this parallel decoding in the experiments section.
3.2.3 Computational complexity of HQ-Transformer
Suppose that we have the top code t ∈
Z l × l and the bottom code b
∈
Z rl × rl , where r is a scaling factor for the top code quantizer.
We denote L top =
l 2 as the length of the top codes and
L local = 1 + r 2 as the number a single top code and all bottom codes associated with the top code.
The time complexities of input embedding, main, and prediction head transformer are then
O ( N in L top L 2 local ) , O ( N main L 2 top ) and O ( N head L top L 2 local ) , where N in , N main , and N head are the number of attention blocks.
And, O ( N main L 2 top ) dominates the other terms, since N main is much larger than N in and N head .
Suppose that the code map resolution of VQ-GAN is equal to the bottom code resolution of HQ-Transformer.
Computational complexity of HQ-Transformer is mainly O ( N main L 2 top ) , which is much lower than the complexity of VQ-GAN O ( N main L 2 top r 4 ) . [Table] [Table] [Figure]
The image visualizes an image reconstructed from the extracted top codes only.
Given the top codes, HQ-Transformer samples bottom codes visualized at the ﬁrst row in Figure 4b and generates realistic images with various details at the second row in Figure 4b
Experiments
In the experiment section, we evaluate our models on class- and text-conditional image generation tasks.
We observe that HQ-Transformer achieves competitive performance to other types of generation models, while allowing high throughput in sample generation.
We also present various ablation studies on our architecture to support the rationale of our choice.
4.1 Conditional Image Generation
Settings We train our models on 1.2M samples from ImageNet [6] trainset for a class-conditional image generation task.
In case of a text-conditional task, we use 15M image-text pairs constructed by
Conceptual Caption (CC) [32] and Conceptual-12M [2].
In training phase of both cases, we resize the short axis of an image to 256 and randomly crop the image to acquire an image of 256 ×resolution.
In testing, we also resize the short axis to 256 and crop the image at the center.
We setup HQ-VAE by f = 8 and r = 2 to produce the top and bottom codes of 8 × 8 and 16 ×resolution, respectively.
We instantiate three types of HQ-Transformer: (a) L main = 12 , and L head =for the smallest model,
(b) L main = 24 and L head = 4 for the middle-size model, and (c) L main = 42 , [Table] [Table]
[Table] and L head = 6 for our largest model. In all cases, L in = 1 and other parameters related to the network size are the same.
We describe more details on the network architecture in the supplementary material.
For evaluation measure, we use Fréchet Inception Distance (FID) [14], precision & recall [31] in class-conditional generation, and FID and CLIP-score [23] in a text-conditional generation task.
We measure throughput in sample generation on a single Tesla A100 in all cases.
Class-conditional image generation Table 1 shows that our largest model, HQ-Transformer (L), achieves the best FID 7.15 compared to two-stage AR models on the similar sequence length, VQGAN ( 16 × 16 ) and RQ-Transformer ( 8 × 8 × 4 ).
We observe that HQ-Transformers report the highest throughput in sample generation than that of other models, since ours generate r 2 bottom codes in parallel given the associated top code.
Compared to ADM, our smallest model (S) can generate samples of similar quality, while achieving 650 × larger throughput.
We also conduct rejection sampling by using ResNet-101 [13] with acceptance rate 0.25 for all our models, and the results are marked in the parenthesis in Table 1.
Here, our models show competitive performance to the other methods.
Figure 4 demonstrates top code-conditional image generation to show the disentanglement of top and bottom codes.
Contrary to typical generative models, HQ-Transformer restricts the diversity of generated images within details by sampling only bottom codes from HQ-Transformer with ﬁxed top code.
Text-conditional image generation Table 2 reports FID and CLIP-score of several models measured on CC3M validation split, and throughput in sample generation.
This shows that our model is comparable to RQ-Transformer in terms of FID (12.86 vs. 12.33) and CLIP-score (0.26 vs. 0.26), while achieving 18% higher throughput than that of RQ-Transformer.
This result shows that our model behaves similarly regardless of data modality.
4.2 Detailed Experiments on HQ-VAE and HQ-Transformer
Comparison with the other stage 1 methods Table 3 compares two types of HQ-VAE, ( f = 16 , r = 2) and ( f = 8 , r = 2) , with the existing stage 1 models, where we measure FID between the original and reconstructed image, referred to as rFID.
We also measure the number of codes activated in reconstruction, and report it in the last column.
rFID from HQ-VAE ( 8 × 8 + 16 × 16 ) is better than the ones from baselines with the similar sequence length:
RQ-VAE ( 8 × 8 × 4 ) and RQ-VAE ( 8 × 8 × 5 ).
In addition, HQ-VAE ( 16 × 16 + 32 × 32 ) achieves better than that of ViT-VQGAN ( 32 × 32 ), while the efﬁcient Transformer can be implemented on the hierarchical code maps, unlike
ViT-VQGAN.
Ablation study on HQ-VAE Table 4 supports the rationale of our choice, showing that resampling with pixel-shufﬂe and -unshufﬂe outperforms the one with average pooling and nearest interpolation by 0.53 in FID, since our choice prevents from loosing information when downsampling.
We also observe that longer training boosts rFID performance from 3.14 to 2.61. We instantiate learnable [Table]
resampling operations with convolution and strided convolution layers with kernel size 2 and stride 2. To avoid top code collapse, we apply additional perceptual loss on top code only reconstructed images. HQ-VAE with perceptual loss achieves reconstruction FID 3.72 on ImageNet validation split. This performance is worse than HQ-VAE with average pooling and pixel-shufﬂe reported in Table 4.
Refer the detailed analysis on the appendix. Ablation study on HQ-Transformer Table 5 compares decoding policies and highlights proposed parallel decoding. VQ-VAE 2 [28] style decoding generates top codes in advance and bottom codes conditioned on top codes strictly hierarchical order. Two standard transformers are utilized in this policy for top code generation and top conditioned bottom code generation.
Three types of locally hierarchical policies are listed.
Causal decoding from RQ-Transformer [18] generates top code and related local bottom codes in a sequential manner. Parallel decoding indicates our procedure; the top code is ﬁrst generated and related r 2 bottom codes are decoded in parallel.
Locally hierarchical decoding policies outperform strictly hierarchical decoding policy because the full visual information can be recovered by combining both top and bottom codes. We observe that our parallel generation scheme achieves better FID than a locally causal decoding, since error accumulation during sampling is alleviated by our parallel decoding approach.
We also validate the effects of soft-label as an alternative to one-hot label in cross-entropy loss function for training HQ-Transformer as proposed in RQ-Transformer [18]. In our case, using soft-labels with temperature τ = 1 improves the performance similarly observed in [18].
Multi-level extensions Multi-level extensions of our approach have the potential to improve the quality of image generation.
Compared to the two-level architectures with the {+same top code resolutions, three-level architectures outperform their two-level counterparts. Adding 
 × 16 bottom codes on ( 4 × 4 + 8 × 8 ) signiﬁcantly boosts FID+} performance from 37.59 to 15.66.
The model with ﬁne three-level codes ( 8 × 8 + 16 × 16 + 32 × 32 ) also outperforms its two-level counterparts. However, we could not achieve the best performance using the three-level architecture with code map resolution ( 8 × 8 + 16 × 16 + 32 × 32 ). Contrary to two-level architecture, longer training of three-level HQ-VAE does not improve the performance of both HQ-VAE and HQ-Transformer. We conjecture that it is because of lower usage of top codes in three-level HQ-VAE as shown in Table 6.
We introduce a two-stage auto-regressive model (HQ-VAE and HQ-Transformer) to generate a novel sample in a locally hierarchical manner extending typical two-stage auto-regressive models.
HQ-VAE represents an image by a top and bottom codes with multiple resolutions, encoding the overall high-level structure of an image in the top codes of low-level and detailed information in the bottom codes of high resolution.
HQ-Transformer models these top and bottom codes efﬁciently and generates novel images of high quality.
Experiments conﬁrm that our approach is capable of generating high-quality samples compared to recent architectures, while achieving high throughput in sample generation. A multi-level extension of our approaches shows potential advantages for higher-resolution image generation.
We expect that tackling lower top code usage in multi-level HQ-VAE leads to better generation performance with faster sampling speed. Acknowledgement This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
[No.2022-0-00959, (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making; No.2022R1A5A708390811,
Trustworthy Artiﬁcial Intelligence]
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We discuss the limitation of our models on the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss it in the supplementary material (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We report in both the paper and the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We report in both the paper and the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The evaluation protocol for conditional generative models does not report the error bars.
Instead, we ﬁx the random seed and make experiments reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We report in both the paper and the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite and report in the supplementary material.
(b) Did you mention the license of the assets?
[Yes] We report in the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We adopt and modify the encoder and the decoder architecture from VQ-GAN [9].
Contrary to VQ-GAN, we set the stride of the ﬁrst convolution layer in the encoder and the last transposed convolution layer in the decoder as two, instead of one in the original implementation as mentioned in the main paper. The encoder takes an image x ∈ R 256 × 256 × 3 and returns encoded feature map z ∈ R 256 f × 256 f × 256 .
{+We adopt HQ-VAE equipped with pixel-unshufﬂe and -shufﬂe for resampling. HQ-VAE with ( f = 16 , r = 2) has a code map resolution 8 × 8 + 
 × and ( f = 8 , r = 2) has code map resolution 16 × 16+32 × 32 .+}
We use HQ-VAE ( f = 16 , r = 2) for all experiments, which gives concise visual codes for efﬁcient training of HQ-Transformer.
We train HQ-VAE ( f = 16 , r = 2) for 50 epochs on ImageNet training split.
We use Adam optimizer with ( β 1 , β 2 ) =
(0 . 9 , 0 . 5) and batch size 128 .
We set a learning rate 4 × 10 − 5 with warm-up 0.
epochs starting from zero learning rate.
Loss weights for reconstruction, perceptual, codebook, and discriminator are set to 1.0, 1.0, 1.0, and 0.75 respectively.
A.2 HQ-Transformer
Overall hyperparameters for HQ-transformer are shown in Table A1.
d e and n e denote the embedding dimension and the number of heads for the transformer.
We use the same notations for parameters in the main paper.
Input embeddings for the main transformer start with start-of-sentence (SOS) tokens.
Class-conditional image generation uses class-conditional learnable embedding as a single SOS token.
Text-conditional image generation uses embeddings of a sequence of text tokenized by a byte pair encoding [ ? ? ].
The length of text tokens is at most 64 text tokens.
We train HQ-Transformer 100 epochs on ImageNet training split and 20 epochs on CC-15M, aggregated training splits of CC-12M and CC-3M. Training HQ-transformer (S), (M), and (L) on ImageNet takes 35, 45, and 75 minutes per epoch with four NVIDIA Tesla A100.
Training HQ-transformer (S) on CC-15M takes 452 minutes per epoch with four NVIDIA Tesla A100.
We use HQ-VAE trained on ImageNet for experiments on class- and text-conditional image generation.
We use AdamW with ( β 1 , β 2 ) =
95) , batch size 512 and weight decay 1 × 10 − 4 .
We adopt two policies for learning rate schedules, warm-up, and cosine annealing.
At the ﬁrst epoch, learning rate is warm-up gradually from lr init = 1 × 10 − 5 to lr peak .
In the remaining training epochs, we use cosine annealing from peak learning rate lr peak to zero.
In ImageNet, we use lr peak = 5 × 10 − 4 .
In CC-15M, we use lower learning rate lr peak = 3 × 10 − 4 and gradient clipping of 1.0 to avoid unstable training.
We select sampling hyper-parameters that achieve the best FID for the largest model for each benchmark.
We use top- k = 2048 and softmax with temperature t = 0 .
95 for sampling in ImageNet if the decoding parameters are not speciﬁed.
For rejection sampling in ImageNet, we use softmax with temperature t = 0 .
95 with acceptance rate 0 .
In CC-3M validation, we use softmax with temperature t = 0 .
9 for sampling.
The performances with respect to sampling-related parameters (top- k , softmax temperature, acceptance ratio) are shown in Figure A6 and Figure A7.
A.3 Libraries and code repository
Our implementation is based on PyTorch 1.10 2 with CUDA 11.3 and PyTorch Lightning 1.5.7 3 .
Major routines of our codes are based on the code repositories for VQ-GAN 4 , minDALL-E 5 and
RQ-Transformer 6 .
B Ablation study B.1 Details of resampling layers in HQ-VAE
We show the detailed analysis of resampling operation on HQ-VAE and its effect on HQ-Transformer in Table A2.
We observe that both reconstruction FID (rFID) and top code only rFID are important for achieving good performance on stage 2.
HQ-VAE with learnable resampling operations can be an example.
We tested two additional loss functions, reconstruction and perceptual loss, on top code only reconstructed image for learnable resampling operations case.
HQ-VAE with pixel reconstruction loss has lower rFID but high top code only rFID.
It results in bad image generation performance compared to that of perceptual loss.
HQ-VAE with pixel-shufﬂe achieve comparable performance on two rFIDs
and it veriﬁes the best generation performance supported by pixel-shufﬂe.
An example of top code collapsing for learnable resampling operations is shown at the top of
Figure A1.
Note that training with the learnable convolution layer is much slower than training with a non-learnable layer.
HQ-VAE with the above learnable resampling layer takes about 300 minutes per epoch, while HQ-VAE with pixel-shufﬂe takes about 170 minutes per epoch with four NVIDIA
Tesla A100 GPUs.
B.2 Input Embedding of HQ-Transformer
Ablation study (a) in Table 5 analyzes the effects on input embedding.
We test two types of input embedding: adding and input embedding transformer (IET) in Eq (10). We introduce a simple adding [Table] [Table]
baseline for input embedding.
Input embedding by adding is fusing embeddings for top and bottom codes and position embedding for top code.
Channel-wise concatenation for embeddings of bottom codes is employed to reduce the parameter size and exploit the residual design where a single top code is aligned to r 2 bottom codes.
[Equation]
However, we think that channel concatenation for bottom codes might cause information loss because adding embeddings of top code and concatenated embeddings of bottom codes requires a smaller channel dimension for each bottom code.
This is reason why we introduce IET, which has the equivalent channel dimension for both top and bottom codes.
We use L in = 1 , 2 for the number of self-attention blocks in IET.
We observe that a single layer for IET outperforms the other settings on FID and recall.
B.3 Prediction Head Transformer (PHT)
We propose parallel decoding for PHT and it assumes conditional independence among bottom codes given top code.
As shown in Figure A2, typical causal attention observes the previous states only.
So, it predicts code in a sequential manner.
But, parallel decoding predicts a single top code and then predicts all bottom codes using contexts from the top colored in the red as shown in mixed attention of Figure A2.
Ablation study (b) in Table A3 shows the effects on attention type for prediction head transformer (PHT).
Our parallel decoding has merit over causal attention in terms of speed.
We measure the throughput of our models with diverse batch sizes as shown in Figure A5.
Throughput for bidirectional decoding outperforms the others and throughput of causal decoding is the smallest one.
Throughput of parallel decoding (mixed attention) is in the middle of two attention types.
As the batch size becomes larger, the throughput gap between prediction head attention types becomes smaller.
[Table]
[Figure]
[Figure]
+ +
[Figure]
B.4 Soft-labeling and softmax temperature in HQ-Transformer
Ablation study (c) in Table A3 examines the effect on soft-label and softmax temperature.
As mentioned in the main paper, soft-label improves FID performance.
We also observe that softmax temperature values below 1.0 improve FID performance if HQ-Transformer is trained with the softlabeling scheme.
The performances with respect to diverse parameters (top- k , softmax temperature, accentance ratio) are shown in Figure A6 and Figure A7.
B.5 Multi-level extension of HQ-VAE and HQ-Transformer
We extend two-level HQ-VAE into three-level and show the disentanglement of top and middle and bottom codes.
The architecture of three-level HQ-VAE is shown in Figure A3 (left).
Disentanglement in three-level HQ-VAE is visualized in Figure A4.
It shows straightforward multi-level extension of
HQ-VAE.
Locally hierarchical autoregressive model for three-level codes—top, middle and bottom codes— is also easily extended from two-level case.
We introduce an additional conditional independence assumption among bottom codes given the associated top and middle codes for the multi-level extension as shown in the equation below.
This multi-level extension of Eq 6 is simply given by applying the chain rule to Eq 5 with the middle code.
[Equation]
where, b i = ( b ( i, 1) , · · · b ( i, 4 r 2 ) ) and, m i =
( b ( i, 1) , · · · b ( i,r 2 ) ) HQ-Transformer with three-level code is easily implemented by plugging in a mixed attention mask for three-level parallel decoding in
HQ-Transformer in Figure A3 (right).
[Figure]
[Figure]
Note that we use decoding parameter top- k = 2048 and temperature (0 . 8 0 , 0 . 8 1 , 0 . 8 2 ) for three-level with code resolution ( 4 × 4 + 8 × 8 + 16 × 16 ) and temperature (0 . 9 0 , 0 .
9 2 ) for three-level with code resolution ( 8 × 8 + 16 × 16 + 32 × 32 ).
Setting low temperature for lower-level of codes improves the performance of three-level case.
However, it does not improve the performance of two-level case.
C Additional image generation examples
We give additional samples for class-, text- and top code-conditional image generation.
Figure A shows examples of class-conditional image generation Figure A9 shows examples of text-conditional image generation.
The ﬁrst row shows reconstructed images from the top code only.
The second row shows reconstructed images from the bottom code only.
The third row shows the ﬁnal resulting images reconstructed from both the top and bottom code.
Generated samples in each class and each prompt are acquired by ﬁxing a random seed as 0.
Based on generated samples from class- and text-conditional image generation, we also conduct top code-conditional image generation shown in Figure A10 and Figure A11.
Figure A10 shows top
[Figure]
[Figure]
on CC-3M validation split measured by FID and CLIP-score with diverse set of (top- k , softmax temperature).
code-conditional image generation by both class- and text-conditional generated images.
Figure A shows 23 generated images from only the single given top code, which demonstrates the diversity in detail generation of our models.
D Limitations and Negative Societal Impacts D.1 Limitations
Two-stage auto-regressive (AR) models have some artifacts in a small region with details.
Figure A shows failure cases due to a scene with a small size of the face.
Synthesized images only from top code in Figure A12 contain the solid color of the face rather than the rough structure of a human face, such as mouth, nose, or eyes.
Solid color on the face without the structure might degrade details created with the independence assumption.
To solve this problem, more information should be injected into the top code without distortion even in the small region.
Note that other two-stage
AR models [9, 18, 28] also experience the mentioned problems as reported in their supplementary materials.
We think that introducing expressive visual codes is still a promising solution to this problem.
Multi-level extensions of our approaches have limitations over two-level baselines.
First, multi-level HQ-VAE has lower effective top code usage, which results in poor performance compared to two-level counterparts. Second, the tuning on parallel decoding parameter is needed. It implies that we need better parallel decoding policy for multi-level extension.
D.2 Negative Societal Impacts
Unknown biases or unﬁltered inappropriate samples in large-scale datasets might contribute to the generation of biased or harmful content and our method also shares the core of these problems.
We believe that more research is needed on those problems for large-scale image generation.
[Figure]