Locally Hierarchical Auto-Regressive Modeling for Image Generation
Abstract
We propose a locally hierarchical auto-regressive model with multiple resolutions of discrete codes.
Our model represents an image with a pyramid of codes via Hierarchically Quantized Variational AutoEncoder (HQ-VAE) in the first stage, and disentangles the information contained in the multi-level codes. For an example of two-level codes, we create two separate pathways to carry high-level coarse structures of input images using top codes while compensating for missing fine details by constructing a residual connection for bottom codes.
An appropriate selection of resizing operations for code embedding maps enables top codes to capture maximal information within images and the first stage algorithm achieves better performance on both vector quantization and image generation.
Hierarchically Quantized Transformer (HQ-Transformer) in the second stage processes a sequence of local pyramids, which consist of a single top code and its corresponding bottom codes. Contrary to other hierarchical models, we sample bottom codes in parallel by exploiting the conditional independence assumption on the bottom codes.
This assumption is naturally harvested from our first-stage model, HQ-VAE, where the bottom code learns to describe local details.
On class-conditional and text-conditional generation benchmarks, our model shows competitive performance to previous AR models in terms of fidelity of generated images while enjoying lighter computational budgets.
Introduction
Deep generative models [1, 2, 3] have become the fundamental basis for synthesizing diverse images of high-resolution from various forms of input supervision, including class conditions [2, 4], text prompts [5, 6] and segmentation maps [7] to name a few.
Most approaches in this paradigm are roughly categorized into two groups depending on whether they explicitly maximize data likelihood or not.
GANs implicitly estimate underlying data distribution to generate novel and crispy samples, but it typically suffers from a lack of diversity [1].
In contrast, it is widely accepted that likelihood-based models are capable of generating diverse examples, compared to GANs, since they aim to cover all modes of a data distribution [8].
This work focuses on auto-regressive models among many likelihood-based approaches since their scalability is validated in several works [5, 9].
[Figure]
Auto-Regressive (AR) models factorize a data likelihood into a product of conditionals in the predefined generation order.
Since AR models already have mode-coverage property, we only need to make minimal changes in architectures and optimization methods for applying to various data modalities, such as audio, natural language, and images.
However, regardless of the modality, training
AR models in a high-dimensional space is computationally expensive since each conditional needs to consider many past observations.
In the context of image generation, VQ-VAE [10] introduces a two-stage learning process, where the first stage represents an image by a discrete code map while the second stage constructs AR priors [2, 11] on discrete sequences.
To make this two-stage framework effective, the codes have to hold sufficient information for high-quality reconstruction of original images from the discrete sequence while the sequence length should be reasonably short for efficient training of AR models.
In this study, we postulate that leveraging the multi-scale structure is a simple but effective way for representing an image as discrete codes based on AR modeling.
We aim to disentangle the information residing in the hierarchy and design an efficient model to generate unseen images by exploiting the multi-level structure maximally.
To achieve this goal, we propose locally hierarchical auto-regressive modeling by extending two-stage AR modeling.
We first extend VQ-VAE to produce a two-level pyramid of discrete codes, where the top code contains high-level coarse information and the bottom encodes the low-level fine details.
Using these locally hierarchical discrete representations, our efficient AR model generates a novel example.
Specifically, the top codes are sequentially generated in a predefined order, and multiple bottom codes associated with a top code are decoded in parallel.
The main contributions of the proposed model are summarized below:
• We propose HQ-VAE (Hierarchically Quantized Variational Auto-Encoder), which encodes an image using a pyramid of two-level codes through hierarchical vector quantization.
To improve the utility of the two-level codes, we disentangle the information contained in the codes, where top codes capture high-level structural information in images while bottom codes carry the remaining low-level details.
• We introduce HQ-Transformer (Hierarchically Quantized Transformer), which is an AR model to generate images based on the hierarchical discrete codes.
We predict a sequence of locally hierarchical codes efficiently by maximizing its likelihood based on the conditional independent assumption imposed on bottom codes.
• The proposed algorithm achieves competitive performance on class- and text-conditional image generation tasks while yielding high throughput in sample generation, compared to various types of generative models.
Figure 1 illustrates several examples of generated images by the proposed algorithm.
We also present the synthesized images based only on top and bottom codes together, which clearly exhibits the proper disentanglement of image contents and low-level details.
Related Works
This section describes existing approaches closely related to our algorithm and highlights the differences between the prior works and ours.
Discrete representations for autoregressive models Learning AR priors based on discrete representations turn out to be effective in language modeling [9, 12].
VQ-VAE [10] pioneers the two-stage image generation process, where the first stage represents an image by a set of discrete codes and the next stage constructs AR priors [2, 11] on the discrete sequences.
Following VQ-VAE, many approaches aim to further reduce the sequence length and improve the architecture [13, 14] or training objectives [7] of the first stage.
Specifically, VQ-GAN [7] shortens the sequence length by using the perceptual and adversarial losses and encoding the semantic information in the codes maximally.
However, these approaches still exhibit the fundamental trade-off between code sequence length and quality of generated images due to a lack of expressiveness of the single-scale vector quantization with the reduced length of code sequence. Hierarchical representations for
models The construction of hierarchical codes is an effective way to control the fundamental trade-off related to the length of code sequence. There exist several image representation and generation methods based on hierarchical discrete codes [8, 15, 16].
{+VQ-VAE-
 [8]+} and HAM [15] make use of the hierarchical relationship between the codes for unseen image generations in a coarse-to-fine manner.
In particular, to tackle the tradeoff issue, VQ-VAE-2 [8] allocates a heavier model for processing coarse top codes and a lighter model for dealing with fine bottom codes.
RQ-Transformer [17] encodes an image into a stack of code maps by iteratively quantizing residual vectors between latent features and code embeddings, effectively reducing the spatial resolution of the code maps without degrading reconstruction quality.
Hierarchical VQ-VAE [16] encodes structural and textual information of an image using different types of codes by training two separate VQ-VAEs, where the first learns with the original images and the second employs the residuals between the original and reconstructed images.
The main idea of Hierarchical VQ-VAE is about how to disentangle the information across the code levels but not about how to train AR models based on this set of codes.
Our approach constructs a pyramid of discrete codes and exploits the codes using AR models for generating high-fidelity images and reducing code resolution.
Diffusion models Diffusion models [3, 18, 19] iteratively denoise a sample from white noise by seeing the contexts in a bidirectional manner.
Most diffusion models have been implemented based on the U-Net [20] style architecture, meaning that the multi-scale feature of an image has been naturally incorporated.
However, they suffer from low throughput in the generation process because, even though the inference requires a lot of iterative steps, the intermediate computation results between the denoising steps are not reusable, unlike AR models.
Locally Hierarchical Auto-Regressive Models
This section describes the details of the proposed method for image generation.
Our approach exploits two levels of codes—top and bottom—for the representation of an image contrary to existing twostage approaches based on AR models.
The proposed architecture encourages the top code to model the overall structure of an image while facilitating the bottom code to complement the remaining details of the image.
To this end, we propose Hierarchically Quantized Variational AutoEncoder (HQ-VAE) for the image vector quantization in the first stage followed by HQ-Transformer to learn the distribution over a sequence of the top and bottom codes in the second stage.
HQ-Transformer achieves better perceptual quality and sampling speed using the underlying hierarchical structure.
Figure 2 illustrates the two-stage procedure of our algorithm based on an AR model.
3.1 Stage 1: Hierarchically Quantized VAE (HQ-VAE)
We present the architecture and the optimization method of the proposed HQ-VAE, for which we first review the vector quantization technique and then discuss our hierarchical quantization module.
E n c o d e r
STAGE 1 (HQ-VAE)
Downsample
Quantizer
Upsample
QuantizerD
e c o d e r e top e bot
Image (256x256)
Reconstruction
STAGE 2 (HQ-Transformer) sos
Main Transformer uuu n h 2 h 1 hh n
Input Embedding Transformer t n b (n,1) b (n,2) b (n,3) b (n,4)
Prediction Head Transformer t n+1 t n+1 t n+1 t n+ b (n+1,1) b (n+1,2) b (n+1,3) b (n+1,4) t n+
Causal Attention Blocks
Bidirectional Attention Blocks
Mixed Attention Blocks t 1 tt 3 tb (1,1) b (1,2) b (1,3) b (1,4)
b (2,1) b (2,2) b (
2,3) b (2,4) b (3,1) b (3,2) b (3,3) b (3,4)
[Figure] [Figure]
3.1.1 Notations for vector quantization
Vector quantization [21] maps a d -dimensional continuous feature vector to a discrete code, which is given by identifying the nearest code embedding of the feature vector in a codebook.
The codebook is a collection of tuples C = { ( s, e s ) } , where e s ∈ R d e denotes the embedding of code s and n C is the size of the codebook C .
Supposing that z ij is a d -dimensional vector at the spatial location ( i, j ) in a feature map z ∈ R l × l × d , a code s ij is given by
[Equation]
where C [ · ] returns the code embedding of an input code and Q ( · ; C ) denotes the quantization process based on a codebook C .
Note that, since every element in a feature map has its own code, the spatial resolution of the code map, denoted by s , is the same as that of the feature map.
3.1.2 Hierarchical vector quantization
Our approach adopts a hierarchical vector quantization scheme to encode input data using two levels of discrete codes—top and bottom denoted by t and b , respectively.
Ideally, the combination of the two-level codes is expected to maintain the full representation power of an original feature map.
In addition, we desire the two kinds of codes to have sufficiently disentangled and heterogeneous information; for example, top codes hold the high-level concept of inputs while bottom ones represent remaining low-level details. To these ends, our network creates two separate pathways designated for the top and bottom codes and makes the branch for bottom codes constrained to have residual information.
In specific, our hierarchical vector quantization transforms a feature map z ∈ R rl × rl × d into a tuple of two code maps ( t , b ) , where t ∈ Z l × l and b ∈
Z rl × rl with an integer scaling factor r ∈ { 1 , 2 , . . . } .
Our scheme first captures the high-level information of a feature map by quantizing its downsampled version using the top codes and computing their embeddings as follows:
[Equation]
where C top denotes the codebook of top codes and Downsample ( · ) is a generic pooling operation to reduce the spatial resolution of a feature map.
Given the top code map t , the bottom codes and its embeddings are derived by
[Equation]
o r
i g i na l
Avg.
Pool
Quantizer
Nearest Interp.
Quantizer
Encoder
Decoder 16 x 16 x K 8 x 8 x K 16 x 16 x K - (a) Average pooling and nearest interpolation o r i g i na l
PixelUnShuffle
Quantizer
PixelShuffle
Quantizer
Encoder
Decoder16 x 16 x K 8 x 8 x 4K 16 x 16 x K (b) Pixel-unshuffle and pixel-shuffle where C bot is the codebook of bottom codes and Upsample ( · ) is any upsampling operation to increase the spatial resolution of a feature map.
Figure 2(left) illustrates this hierarchical quantization procedure.
Note that the residual vector between the original feature map and the upsampled top code embedding map, denoted by z bot , involves sampling and quantization errors.
The bottom codes with their embeddings should compensate for the errors and improve the reconstruction quality of the original feature map when combined with the top-level information. One question is which operations are appropriate for Downsample ( · ) and Upsample ( · ) . We observe that non-learnable layers give favorable solutions for the resizing operations compared to naïve learnable counterparts.
Straightforward choices for those operations are the average pooling and the interpolation with nearest neighbors.
Despite the reasonable performance given by this strategy, Figure 3a exhibits that the overall structure reconstructed by the top code embeddings is below expectation.
Instead, it turns out that a combination of pixel-unshuffle and pixel-shuffle operations [22] effectively push more information into the top codes while making the bottom codes focus on local details.
Given the feature map z ∈ R rl × rl × d and a scaling factor r , the pixel-unshuffle operation returns a tensor in the shape of l × l × r 2 d encoding high-level information of the original feature map.
To compute the residual, we pixel-shuffle the top code embedding and restore the original dimensionality of z .
Figure 3 clearly demonstrates that the top codes given by the pixel-unshuffle followed by the quantization hold the primary information of an input image more than the ones given by the average pooling.
3.1.3 HQ-VAE: architecture and optimization
Overall structure HQ-VAE consists of three parts: encoder E ( · ) , decoder D ( · ) , and hierarchical quantizer H ( · ; r ) , as shown in Figure 2(left), where r is the additional scaling factor for the top code.
Given an image x , an encoder E ( · ) extracts the feature map z .
Then, the hierarchical quantizer H ( · ; r ) is applied to the feature map, providing the top and bottom codes ( t , b ) and their associated code embeddings: e top and e bot .
Now, the decoder D ( · ) takes the top and bottom code embeddings to reconstruct the input image.
Objective function To train three parts in HQ-VAE, our objective function is composed of the reconstruction loss introduced in VQ-VAE [10] as well as adversarial and perceptual losses to improve the fidelity of reconstructed images [7].
Here, we only present the reconstruction loss as below:
[Equation]
where sg ( · ) refers to a stop-gradient operation, and β is set to 0.25 as suggested in [10].
The first term measures ℓ 2 distance between the reconstructed and original images.
The second term enforces both the top and bottom code embedding to be located close to the inputs of the quantizers.
To update the codebook, we make use of exponential moving average (EMA) to accelerate convergence as discussed in [10].
Implementation details We adopt the same network structures of the encoder and the decoder as
VQ-GAN [7] except two minor modifications. To increase throughput in training, we set the stride of the first convolution layer in the encoder and the last transposed convolution layer in the decoder as 2, instead of 1 in the original implementation.
We replace the batch normalization [23] in the discriminator by the group normalization [24] to achieve consistent results regardless of batch sizes.
3.2 Stage 2: Hierarchically Quantized Transformer (HQ-Transformer)
This section presents HQ-Transformer (see Figure 2(right)) to autoregressively model the top and bottom codes by fully exploiting their spatial relationships obtained from the first stage.
3.2.1 Formulation
Our objective is to maximize the log-likelihood of a tuple of t and b .
Given the scaling factor r for the top, the resizing operation aligns a single top code t i with r 2 bottom codes b ( i,k ) , where 1 ≤ k ≤ r 2 .
Figure 2 illustrates an example of top and bottom codes with r = 2 (in the input embedding transformer part), where all bottom codes aligned with a single top code t i are b i =
( b ( i, 1) , b ( i,r 2 ) ) .
The likelihood of the top and bottom codes is given by
[Equation]
where the inequality notation <
i denotes a collection of all indices less than i .
The conditional is further factorized into
[Equation]
The last term in (6) is maximized by predicting the current position given the previous top and bottom codes.
By assuming that the bottom codes are conditionally independent given the associated top code, the first term of the right-hand side is formulated more efficiently as follows:
[Equation]
Note that the assumption we made here is reasonable because the bottom codes are supposed to contain local details while the top codes are enforced by our stage 1 model, HQ-VAE, to encode a global landscape.
3.2.2 Model
Similar to GPT-like architectures [9, 25], HQ-Transformer consists of three sub-modules based on the transformer architecture, which are referred to as Input Embedding Transformer (IET), Main
Transformer (MT), and Prediction Head Transformer (PHT), as illustrated in Figure 2(right). IET generates an embedding to be used as an input for MT, and the hidden state given by MT is fed into PHT to generate top and bottom codes, which are given back to IET in the next step. We discuss each of the three modules in the rest of this section.
Input Embedding Transformer (IET) IET transforms a tuple of ( t i , b i ) into a single embedding vector u i , ∀ i .
To implement this, we first construct a local embedding by summing up the code embedding and the positional encoding, which is given by [Equation] for
[Equation]
where e top,IET i and e bot,IET ( i,k ) are the learnable code embedding of t i and
b ( i,k ) , respectively, and PE top [ · ] and PE bot [ · ] denote the learnable positional encodings for the top and bottom 2 .
Note that the code embeddings are learned within this IET module, by which we construct the codebooks C top,IET and C bot,IET .
These local embeddings are refined by a bidirectional transformer,
[Equation]
i [Equation] where the structure is the same as GPT-2 [9] and we employ a single GPT-2 block composed of a selfattention followed by an MLP block.
Then, the final input embedding u i for the main Transformer block is simply given by averaging all updated local embeddings as
[Equation]
Main Transformer (MT)
This module updates the outputs of IET by a stack of N main causal self-attention and MLP blocks:
[Equation]
where the sequence length is equal to the number of top codes and the start-of-sentence token (SOS) is employed as the first input of the MT ( · ) function.
This function also has the same structure as GPT-2.
Prediction Head Transformer (PHT)
This module predicts the top and bottom codes from the hidden representation derived by the main transformer, where the bottom codes are generated in parallel based on their associated top code and a hidden representation from MT.
In specific, we first define the input sequence of PHT as below: [Equation] if
[Equation]
where e top,PHT i denotes the learnable code embedding of t i defined within PHT.
This sequence is then updated to produce the conditional probabilities of the top and bottom codes as follows:
[Equation]
where each output probability is defined as
[Equation]
Note that we predict the bottom code distributions in parallel. In PHT, we employ a self-attention block of a mixture of the positions of a top code and multiple bottom codes, which is conditioned on the top position while the attention between bottom positions are unconditional.
Compared to the full autoregressive modeling on the bottoms, this parallel decoding leads to speed-up in sampling and even better image generation quality.
We will discuss its benefits in the experiments section.
3.2.3 Computational complexity of HQ-Transformer
Given the top code t ∈ Z l × l and the bottom code b ∈
Z rl × rl , where r is a scaling factor for the quantization of top codes, the length of the top codes is L top =
l 2 and the number of codes in a local patch is L local = 1 + r 2 because there exist r 2 bottom codes associated with a single top code.
The time complexities of IET, MT, and PHT are then given by O ( N IET L top L 2 local ) , O ( N MT L 2 top ) and
O ( N PHT L top L 2 local ) , respectively, where N IET , N MT , and N PHT are the numbers of attention blocks in individual modules.
The time complexity of MT dominates those of the other two modules because N MT is much larger than N IET and N PHT .
Because the code map resolution of VQ-GAN is equal to the bottom code resolution of HQ-Transformer, the computational complexity of VQ-GAN,
O ( N MT L 2 top r 4 ) , is higher than the complexity of HQ-Transformer, O ( N MT L 2 top ) .
[Table] [Table]
3.3 Sampling from HQ-Transformer and HQ-VAE
For inference
[Table]
we first generate discrete codes using HQ-Transformer and synthesize new images using HQ-VAE based on the codes. We begin with the start-of-sentence (SOS) token and obtain its hidden representation from MT in HQ-Transformer, where the class-conditional SOS is employed in the case of class-conditional generation.
Then, PHT takes the hidden representation as its input embedding at the first step and predicts a top code distribution. We sample a top code from the distribution and forward it to the next step of PHT as to obtain the bottom codes, which are actually acquired in parallel by forwarding the top code embedding to the multiple next steps. The top and bottom codes are given to IET, which outputs the embedding for the input to MT.
We repeat this procedure in an auto-regressive manner until we obtain all the top and bottom codes. Once completing the generation of the codes, we finally produce a synthetic image using the decoder of HQ-VAE based on the discrete codes.
We evaluate our models, referred to as HQ-TVAE hereafter, on class- and text-conditional image generation tasks.
4.1 Conditional Image Generation
Experiment settings We train our models on 1.2M images in the train split of ImageNet [29] for class-conditional image generation.
For text-conditional tasks, we employ 15M image-text pairs in
Conceptual Caption (CC) [30] and Conceptual-12M [31].
In the training phase of both cases, we resize the short axis of an image to 256 and randomly crop it to acquire a 256 × 256 image.
In testing, we also resize the short axis to 256 and crop the image at the center.
The scaling factor in HQ-VAE is set to 2, i.e. , r = 2 , to produce the top and bottom codes of 8 × 8 and 16 × 16 resolution by default, respectively.
We test three versions of HQ-Transformer by varying hyperparameters: (a) L MT = 12 , and L PHT = 4 for the smallest model, (b) L MT =and L PHT = 4 for the mid-size model, and (c) L MT = 42 , and L PHT = 6 for our largest model.
In all cases, L IET = 1 and other parameters related to the network size are fixed.
We describe more details about the network architecture in the supplementary material.
As evaluation metrics, we adopt
Fréchet Inception Distance (FID) [32], precision & recall (PR) [33] for class-conditional generation, [Figure] [Table] and FID and CLIP-score [34] for text-conditional generation.
We measure throughput of sample generation on a single Tesla A100 GPU
Class-conditional image generation Table 1 presents that our largest model, HQ-TVAE (L), achieves the best FID (7.15) compared to other two-stage AR models with the similar sequence length, VQ-GAN ( 16 × 16 ) and RQ-Transformer ( 8 × 8 × 4 ).
We observe that HQ-TVAE report the highest throughput in sample generation than other models because it generates r 2 bottom codes in parallel given the associated top code.
Compared to ADM [26], our smallest model (S) generates images of similar quality while achieving 650 × throughput.
We also conduct rejection sampling using ResNet-101 [35] with acceptance rate 0.25 for all our models and report the results within the parenthesis in Table 1.
Figure 4 demonstrates the qualitative results of our approach, which also include the generated images based only on top codes to visualize the representation disentanglement.
Interestingly.
contrary to typical generative models, HQ-TVAE restricts the diversity of generated images within details by sampling only bottom codes from HQ-Transformer with fixed top code.
Text-conditional image generation Table 2 reports FID and CLIP-score of several models measured on the CC3M validation split, and throughput in sample generation.
The results show that our model is comparable to RQ-Transformer in terms of FID (12.86 vs. 12.33) and CLIP-score (0.26 vs. 0.26), while achieving 18%p higher throughput than RQ-Transformer.
4.2 Analysis of HQ-VAE and HQ-Transformer Models
Comparison with other stage 1 methods Table 3 compares HQ-VAE implementations with two different feature map sizes (shown in the parentheses) from the encoder with the existing stagemodels, where we measure FIDs between the original and reconstructed images, referred to as rFID.
The last column of the table reports the utilization of codebook(s) for the representations of the images in the ImageNet validation dataset, and our algorithm turns out to make full use of the codebooks.
rFID from HQ-VAE ( 8 × 8 + 16 × 16 ) is better than the baselines with the similar sequence length, e.g. , RQ-VAE ( 8 × 8 × 4 ) and RQ-VAE ( 8 × 8 × 5 ).
In addition, HQ-VAE ( 16 × 16 + 32 × 32 ) outperforms ViT-VQGAN ( 32 × 32 ) although we admit that ViT-VQGAN can be implemented more efficiently based on the hierarchical code maps. [Table] [Table]
[Table]
Ablation study on HQ-VAE Table 4 supports the rationale of our choice, showing that resizing with pixel-unshuffle and -shuffle outperforms the average pooling followed by the nearest interpolation by 0.53 in FID since it prevents information loss during the downsampling operation.
We also observe that longer training boosts rFID performance from 3.14 to 2.61.
Note that learnable resizing turns out to be not effective, where we adopt a simple network with single convolution and transposed convolution layers with kernel size 2 and stride 2, for downsampling and upsampling, respectively.
We impose the same loss function on the reconstructed images using only the top codes to avoid top code collapse.
Refer to the appendix for more detailed analysis .
Ablation study on HQ-Transformer Table 5 compares the effect of decoding policies and highlights the benefit of the proposed parallel decoding.
{+VQ-VAE-
+}
[8] first generates the entire top codes and then derives the bottom codes one by one with global conditioning on the top codes. RQ-Transformer [17] generates a top code and the relevant bottom codes sequentially locally conditioned on the top code.
We observe that our parallel generation scheme achieves better FID than the sequential methods since error accumulation during sampling is alleviated by our parallel decoding.
We also validate the effects of soft-label as an alternative to one-hot label in cross-entropy loss function for training HQ-Transformer as proposed in RQ-Transformer [17].
The use of soft-labels with temperature τ = 1 in our case improves performance as similarly observed in [17].
Multi-level extensions Table 6 present that three-level architectures outperform the two-level counterparts.[-Adding 
-]
For these experiments, we train two different combinations of three-level codes { (4 × 4) , (8 × 8) , (16 × 16) } and { (8 × 8) , (16 × 16) , (32 × 32) } .
The results show that the use of three-level codes is helpful for improving performance at the early stage of training (15 epochs for training HQ-VAE). However, they fail to achieve additional gains with more iterations probably due to the lower usage of top codes as shown in the “code utilization" column of the table. Detailed architectures for multi-level extensions are shown in the supplementary file.
Conclusion
We presented a two-stage auto-regressive model, which consists of HQ-VAE and HQ-Transformer, to generate images based on hierarchically quantized codes.
HQ-VAE represents an image by a two-level discrete codes with different spatial resolutions, where it encodes the high-level structure of an image using the top codes and the low-level information in the bottom codes.
HQ-Transformer models the multi-level codes efficiently and the combination of the two modules successfully generates unseen images.
Experiments confirm the capability and potential of our model for high-quality and high-resolution image generation with a fraction of computational cost compared to existing methods.
Acknowledgement This work was partly supported by IITP grant [No.2022-0-00959, (Part 2)
Few-Shot Learning of Causal Inference in Vision and Language for Decision Making, 2021-0-01343, Interdisciplinary Program in AI (SNU)] and NRF grant [No.2022R1A5A708390811, Trustworthy
Artificial Intelligence] funded by the Korean government (MSIT).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We discuss the limitation of our models on the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss it in the supplementary material (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We report in both the paper and the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We report in both the paper and the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The evaluation protocol for conditional generative models does not report the error bars.
Instead, we fix the random seed and make experiments reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We report in both the paper and the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite and report in the supplementary material.
(b) Did you mention the license of the assets?
[Yes] We report in the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[-HQ-VAE with ( f = 16 , r = 2) has a code map resolution 8 × 8 + 
-]
[N/A] (c) Did you include the estimated
hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]