V ECTOR - QUANTIZED I MAGE M ODELING WITH I MPROVED VQGAN
Jiahui Yu
Xin Li
Jing Yu Koh
Han Zhang Ruoming Pang James Qin Alexander Ku
Yuanzhong Xu Jason Baldridge Yonghui Wu Google Research jiahuiyu@ com
A BSTRACT
Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks.
Motivated by this success, we explore a Vector-quantized Image Modeling ( VIM ) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively.
The discrete image tokens are encoded from a learned Vision-Transformerbased VQGAN ( ViT-VQGAN ).
We ﬁrst propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efﬁciency and reconstruction ﬁdelity.
The improved ViT-VQGAN further improves vectorquantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning.
When trained on ImageNet at 256 × 256 resolution, we achieve Inception Score (IS) of 175.1 and
Fr ´ echet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla
VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively.
Based on
ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained
Transformer by averaging intermediate features, similar to Image GPT (iGPT).
This ImageNet-pretrained VIM-L signiﬁcantly beats iGPT-L on linear-probe accuracy from 60.3% to 73.2% for a similar model size.
ViM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.
I NTRODUCTION
Natural language processing (NLP) has recently experienced dramatic improvements from learning general-purpose representations by pretraining language models on unlabeled text corpora.
This strategy has produced large performance gains for a wide range of natural language generation (NLG) and natural language understanding (NLU) tasks (Dai & Le, 2015; Radford et al., 2018; 2019; Brown et al., 2020).
Conceptually, generative pretraining models the data density P ( X ) in a tractable way, with the hope of also helping discriminative tasks of P ( Y | X ) (Lasserre et al., 2006); importantly, there are no limitations on whether the signals are from the language domain or others, such as vision.
In computer vision, in contrast, most recent unsupervised or self-supervised learning research focuses on applying different random augmentations to images, with the pretraining objective to distinguish image instances (Chen et al., 2020b;
He et al., 2020; Chen et al., 2020d; Grill et al., 2020;
Chen et al., 2020c; Caron et al., 2021).
The quality of learned representation relies on manually chosen augmentations, such as random brightness, cropping, blurring, and others.
Chen et al.
(2020a) explored GPT-style (Radford et al., 2018)
generative pretraining on images by autoregressively predicting pixels without incorporating knowledge of the 2D structure.
Each pixel is represented as a 9-bit value created by clustering (R, G, B) pixel values, using k-means with k=512.
Unfortunately, this color encoding does not scale to typical image resolutions as it entails very long sequences to represent the image ( e.g ., 224 × 224 resolution leads to 50 , 176 tokens per image), and this demands much more memory and computation for training, compared to language models.
As a result, iGPT’s maximum resolution is 64 × 64 for image recognition at scale—which severely limits its representation capabilities.
Remarkable image generation results have been achieved by pre-quantizing images into discrete latent variables and modeling them autoregressively, including VQVAE (Oord et al., 2017), DALLE (Ramesh et al., 2021) and VQGAN (Esser et al., 2021).
In these approaches, a convolution neural network (CNN) is learned to auto-encode an image and a second stage CNN or Transformer is learned to model the density of encoded latent variables.
These have been proved effective for image generation, but few studies have evaluated the learned representation in discriminative tasks (Ramesh et al., 2021; Esser et al., 2021).
We explore an approach we refer to as Vector-quantized Image Modeling (VIM) and apply it to both image generation and image understanding tasks.
VIM follows a two-stage approach:
• Stage 1: Image Quantization.
Given an image of resolution 256 × 256 , a VisionTransformer-based VQGAN encodes it into 32 × 32 discretized latent codes where the codebook size is 8192 .
We propose multiple improvements–from architecture to codebook learning–to VQGAN (Esser et al., 2021).
The resulting ViT-VQGAN is more efﬁcient and improves reconstruction ﬁdelity in terms of pixel-wise reconstruction metrics, Inception Score (IS) and Fr ´ echet Inception Distance (FID).
ViT-VQGAN is trained end-to-end on image-only data with combined objective functions of logit-laplace loss, (cid:96) 2 loss, adversarial loss and perceptual loss (Johnson et al., 2016; Zhang et al., 2018).
• Stage 2: Vector-quantized Image Modeling.
We train a Transformer model to predict rasterized 32 × 32 = 1024 image tokens autoregressively, where image tokens are encoded by a learned Stage 1 ViT-VQGAN
For unconditional image synthesis or unsupervised learning, we pretrain a decoder-only Transformer model to predict the next token.
For class-conditioned image synthesis, a class-id token is prepended before the image tokens.
To evaluate the quality of unsupervised learning, we average the intermediate Transformer features and learn a linear head to predict the logit of the classes ( a.k.a ., linear-probe).
We show that one key component for improving both image generation and image understanding with VIM is to have a better image quantizer with respect to both computational efﬁciency and reconstruction quality.
An efﬁcient quantizer can speed up Stage 2 training, where random augmentations are applied ﬁrst to an image, followed by the encoder of image quantizer to obtain the input tokens.
Moreover, an image quantizer with better reconstruction quality can reduce information loss compared with the original image in pixel space, which is critical for image understanding tasks.
The evaluations of our proposed ViT-VQGAN and VIM are studied with three aspects.
(1) We evaluate the image quantizer based on reconstruction quality metrics including (cid:96) 1 distance, (cid:96) 2 distance, log-laplace distance, as well as Inception Score (IS) and Fr´echet Inception Distance (FID) of reconstructed images.
(2) We evaluate the capabilities of the learned quantizer for unconditional or class-conditioned image synthesis based on FID and IS, and compare with other methods.
We rely on linear-probe accuracy to evaluate representations with the common intuition that good features should linearly separate the classes of downstream tasks.
R ELATED W ORK
Image Synthesis.
Image generation has received much attention with the progress of deep generative models, including Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Zhang et al., 2019b), Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Vahdat & Kautz, 2020),
Diffusion Models (Song & Ermon, 2019; Dhariwal & Nichol, 2021) and Autoregressive Models (van den Oord et al., 2016; Parmar et al., 2018).
Unlike many autogressive methods which generate sequence directly in pixel space, VQVAE (van den Oord et al., 2017; Razavi et al., 2019) decomposes the image generation process into two stages: the ﬁrst stage trains a vector quantized autoencoder with image reconstruction objective to convert an image into a shorter sequence of discrete tokens.
Then the second stage learns an autoregressive model, e.g., PixelSNAIL (Chen et al., 2018), to model the underlying distribution of token sequences.
Driven by the effectiveness of VQVAE and progress in sequence modeling (Vaswani et al., 2017; Devlin et al., 2019), many approaches follow the two-stage paradigm.
DALL-E (Ramesh et al., 2021) improves token prediction in second stage by using Transformers (Vaswani et al., 2017), resulting in a strong text-to-image synthesis model.
VQGAN (Esser et al., 2021) further uses adversarial loss and perceptual loss (Johnson et al., 2016; Zhang et al., 2018) to train a better autoencoder in the ﬁrst stage to synthesize greater detail in images.
Image Recognition with Generative Pretraining.
Many image generation models (Goodfellow et al., 2014; Kingma & Welling, 2014; Radford et al., 2016; Donahue et al., 2017; Higgins et al., 2017) have been studied for their capabilities in representation learning.
However, their performance is usually not superior to competing self-supervised approaches that solve auxiliary classiﬁcation tasks (Noroozi & Favaro, 2016a; Gidaris et al., 2018a; van den Oord et al., 2018).
BigBiGAN (Donahue & Simonyan, 2019a) ﬁrst demonstrated that a generation-based model can match other self-supervised methods in representation learning on ImageNet.
iGPT (Chen et al., 2020a) uses the autoregressive objective to learn a giant transformer that directly predicts pixel values, producing even more competitive results.
Compared to iGPT, our method ﬁrst tokenizes the original image into discrete image tokens and then trains a transformer to predict them.
As a result, our approach obtains comparable results with smaller model and less data.
Similar to our method in predicting image tokens, BEiT (Bao et al., 2021) follows pre-training scheme of BERT Devlin et al.
(2019) by learning to recover randomly masked image tokens with a bidirectional transformer.
Unlike BEiT, we explore vector-quantized image modeling for image generation in addition to image recognition.
V ECTOR - QUANTIZED I MAGES WITH V I T-VQGAN
The Vector-quantized Variational AutoEncoder (VQVAE) (van den Oord et al., 2017) is a CNNbased auto-encoder whose latent space is a matrix of discrete learnable variables, trained end-to-end via straight-through estimation.
Esser et al.
(2021) introduce VQGAN, a model which improves upon VQVAE by introducing an adversarial loss produced by a discriminator.
Below, we introduce further improvements to VQGAN that boost efﬁciency and enhance reconstruction quality.
3. VQGAN WITH V ISION T RANSFORMERS
The core network architectures used by both VQVAE and VQGAN to encode and reconstruct images are CNNs.
VQGAN introduces transformer-like elements in the form of non-local attention block (Zhang et al., 2019a), allowing it to capture distant interactions with fewer layers.
We propose taking this approach one step further by replacing the CNN encoder and decoder with Vision
Transformer (ViT) (Dosovitskiy et al., 2020), as shown in Figure 1.
Given sufﬁcient data (for which unlabeled image data is plentiful)
we ﬁnd that ViT-VQGAN is less constrained by the inductive priors imposed by convolutions.
Furthermore, ViT-VQGAN yields better computational efﬁciency on accelerators, and produces higher quality reconstructions, as shown in Table 1.
The encoder of ViT-VQGAN ﬁrst maps 8 × 8 non-overlapping image patches into image tokens, followed by Transformer blocks, encoding a 256 × 256 resolution image into a 32 × 32=1024 token sequence.
The decoder performs the inverse operation, mapping each image token from latent variables back to 8 × 8 image patches and regrouping them into a 256 × 256 image (see Figure 1).
At the output of transformer blocks, we apply a two-layer feed-forward network with a tanh activation layer
[Table]
in turn further in the middle.
No activation is applied at the output of ViT-VQGAN encoder or decoder.
We ﬁnd that this simple approach yields high quality reconstructions without any noticeable grid artifacts.
3. C ODEBOOK L EARNING
Vanilla VQVAEs usually suffer from low codebook usage due to the poor initialization of the codebook.
Therefore, during training a signiﬁcant portion of codes are rarely used, or dead .
The reduction in effective codebook size results in worse reconstructions in stage 1 quantizer training and poor diversity in stage 2 for image synthesis.
As a result, VQGAN (Esser et al., 2021) relies on topk and top- p (nucleus) sampling heuristics (Holtzman et al., 2020) with a default codebook size of 1024 to obtain best results for image synthesis.
We propose two improvements that can signiﬁcantly encourage the codebook usage even with a larger codebook size of 8192 .
During image synthesis, we perform simple sampling with temperature of 1 . 0 without top- k and top- p heuristics.
The training objective of vector-quantization is deﬁned as follows:
[Equation]
Here, sg ( x ) ≡ x, dd x sg ( x ) ≡ 0 is the stop-gradient operator, β is a commitment loss hyperparameter set to 0.25 in all our experiments, and e is the codebook vector.
The quantized codebook index is determined by looking up the codebook vector closest to the input features z e ( x ) in terms of the
Euclidean distance, yielding i = argmin
j (cid:107)
z e ( x ) − e j (cid:107) 22 .
Factorized codes.
We introduce a linear projection from the output of the encoder to a lowdimensional latent variable space for code index lookup ( e.g ., reduced from a 768-d vector to a 32-d or 8-d vector per code) and ﬁnd it has an immediate boost of codebook usage.
The factorization can be viewed as decoupling code lookup and code embedding: we lookup the the closest variable encoded from input on a lower-dimensional lookup space and then project the matched latent code to the high-dimensional embedding space.
Our experiments show reducing dimension of lookup space from 256-d to 32-d consistently improves reconstruction quality.
A detailed illustration is provided in the supplementary materials.
(cid:96) 2 -normalized codes.
We also apply (cid:96) 2 normalization on the encoded latent variables
z e ( x ) and codebook latent variables e .
The codebook variables are initialized from a normal distribution.
By mapping all latent variables on a sphere, the Euclidean distance of (cid:96) 2 -normalized latent variables (cid:107) (cid:96)
( z e ( x )) − (cid:96) 2 ( e j ) (cid:107) 22 evolves to the cosine similarity of two vectors between z e ( x ) and e , further improving training stability and reconstruction quality shown in our experiments.
3. V I T-VQGAN T RAINING L OSSES
We use a combination of logit-laplace loss, (cid:96) 2 loss, perceptual loss (Johnson et al., 2016; Zhang et al., 2018) based on VGG network (Simonyan & Zisserman, 2014) and GAN loss with architecture of StyleGAN discriminator (Karras et al., 2020).
Loss balancing weights are conﬁgured with a hyper-parameter sweep to optimize image reconstruction quality, codebook usage, FID and Inception Score.
After the sweep, we apply the same set of hyper-parameters of training losses to all datasets including CelebA-HQ, FFHQ, and ImageNet.
Logit-Laplace loss can be viewed as normalized (cid:96) 1 loss which assumes the noise at the pixel level is laplace-distributed while (cid:96) 2 loss assumes
[Table]
the noise is of a Gaussian distribution.
We ﬁnd logit-laplace loss contributes to codebook usage while (cid:96) 2 loss and perceptual loss signiﬁcantly contribute to FID.
The ﬁnal loss combination we used by default is L = L VQ + 0 .
1 L Adv + 0 .
1 L Perceptual + 0 .
1 L Logit-laplace + 1 .
L 2 .
One caveat on the VGG-based perceptual loss is that the VGG network is pretrained with supervised classiﬁcation loss, so the supervision might leak into Stage 2 for linear-probe accuracy measurement.
Thus, for all of our reported unsupervised learning results, we exclude the perceptual loss during
ViT-VQGAN training.
For all unconditional and class-conditioned image synthesis, we use ViTVQGAN quantizers trained with perceptual loss, as it leads to higher-ﬁdelity reconstructions.
V ECTOR - QUANTIZED I MAGE M ODELING
With a learned ViT-VQGAN, images are encoded into discrete latent code ids ﬂattened in the raster order, similar to Image GPT (Chen et al., 2020a).
A decoder-only Transformer model is used to model the density of image data P ( x ) autoregressively as
[Equation]
where θ is learnable weights.
The training objective is to minimize the negative log-likelihood of the data L = E x ∈ X
[ − logP ( x )] .
Table 2 summarizes the architecture conﬁgurations for the Transformers.
We ﬁrst embed discrete image token ids into a learnable embedding space at each position, with an additive learnable 2D positional embedding.
Both embedding dimensions are the same as model dimension.
We apply a stack of Transformer blocks to the inputs with causal attention over the entire sequence.
A dropout ratio of 0.1 is used in all residual, activation and attention outputs.
At the ﬁnal layer of all Transformer blocks, we apply an additional layer normalization.
4. I MAGE S YNTHESIS
With a pretrained generative Transformer model, unconditional image generation is achieved by simply sampling token-by-token from the output softmax distribution.
All samples used for both qualitative and quantitative results are obtained without temperature reduction.
The sampled tokens are then fed into the decoder of ViT-VQGAN to decode output images.
Our default Stage 1 ViTVQGAN encodes input images of resolution 256 × 256 into 32 × 32 latent codes with a codebook size 8192 , while Stage 2 Transformer takes the ﬂattened image tokens with total a length of 1024 .
Class-conditioned ImageNet generation is also a widely used benchmark for measuring capabiltiy of models for image synthesis.
We extend the unconditional generation to class-conditioned generation by prepending a class-id token before the image tokens.
Separate embedding layers are learned from scratch for class-id token and image tokens, with the embedding dimension the same as the
Transformer model dimension.
During sampling, a class-id token is provided at the ﬁrst position to decode the remaining image tokens autoregressively.
4. U NSUPERVISED L EARNING
For the image understanding task, we feed all image tokens of the input into a pretrained Transformer, and get a sequence of 1024 token features.
Similar to Image GPT (Chen et al., 2020a),
[Table]
we take a layer output at a speciﬁc block l over total blocks L , average over the sequence of token features (frozen) and insert a softmax layer (learnable) projecting averaged feature to class logits.
We only take one speciﬁc Transformer block output instead of concatenating different block outputs as in iGPT (Chen et al., 2020a).
We ﬁnd that most discriminating feature for the linear-probe is typically near the middle of all Transformer blocks.
E XPERIMENTS 5. I MAGE Q UANTIZATION
We train the proposed ViT-VQGAN on three datasets separately, CelebA-HQ (Karras et al., 2019),
FFHQ (Karras et al., 2019)
[Table]
and ImageNet (Krizhevsky et al., 2012).
For CelebA-HQ and FFHQ, we follow the default train and validation split as VQGAN (Esser et al., 2021).
For Stage 1 image quantization, three different architecture sizes are experimented, Small, Base and Large for either encoder or decoder, as deﬁned in Table 2.
The smallest ViT-VQGAN-SS has a Small-size encoder and Small-size decoder, while ViT-VQGAN-BB has a Base-size encoder and Base-size decoder.
The largest ViT-VQGAN-SL has an asymmetric Small-size encoder and Large-size decoder, with the motivation that Stage 2 training only requires forward propagation of the encoder of ViT-VQGAN (in inference/decoding for image synthesis, the decoder of ViT-VQGAN is still required to decode images from codes predicted during Stage 2).
We train all ViT-VQGAN models with a training batch size of 256 distributed across 128 CloudTPUv4 for a total 500 , 000 training steps.
For both ViT-VQGAN and StyleGAN discriminator, Adam optimizer (Kingma & Ba, 2014) is used with β 1 = 0 .
9 and β 2 = 0 .
99 with the learning rate linearly warming up to a peak value of 1 × 10 − 4 over 50 , 000 steps and then decaying to 5 × 10 − 5 over the remaining 450 , 000 steps with a cosine schedule.
We use a decoupled weight decay (Loshchilov & Hutter, 2017) of 1 × 10 − 4 for both ViT-VQGAN and StyleGAN discriminator.
All models are trained with an input image resolution 256 × 256 on CloudTPUv4.
Table 3 shows FID between reconstructed images and original images in the validation split on
ImageNet, CelebA-HQ and FFHQ datasets.
Without multi-scale hierarchical codebook or gumbelsoftmax, ViT-VQGAN is able to achieve better FID with a large codebook size of 8192 compared with vanilla VQGAN.
Table 4 provides extensive ablations on the proposed modiﬁcations, with empirical results on mean (cid:96) 1 distance, (cid:96) 2 distance, logit-laplace distance, Inception Score and FID on ImageNet.
Among different model sizes, ViT-VQGAN-SS (small-encoder, small-decoder) performs worse than ViTVQGAN-BB (base-encoder, base-decoder) and ViT-VQGAN-SL (small-encoder, large-decoder), but achieves much better throughput.
The CNN-based VQGAN architecture is worse in both quality and throughput compared with ViT-based VQGAN.
The StyleGAN-based discriminator (Karras et al., 2019) is more stable and yields better reconstruction quality than PatchGAN (Isola et al., 2017)
[Table]
D ec od e r
S i
[Table]
A r c h it
[Table]
t u r e
D i s c
r
i m i n a t
o r
L a t e n t
d
i m (cid:96)
[Table]
r m a li
[Table]
e
[Table]
↓ (cid:96)
[Table]
e
[Table]
↓
L og
[Table]
L a p l ace
[Table]
I S ↑
F I
[Table]
↓
C od e book
U s a g
[Table]
↓
T h
[Table] [Table]
(which was used for VQGAN).
For codebook learning, factorized codes with low-dimensional latent variables consistently achieve better reconstruction quality when the latent dimension is reduced from 256 to 16 or 8.
Moreover, removing (cid:96) 2 -normalization leads to much worse results.
5. I MAGE S YNTHESIS
On top of the pre-learned ViT-VQGAN image quantizer, we train stage 2 transformer models for unconditional and class-conditioned image synthesis and compare with previous work.
We use a default model size of ViT-VQGAN-SS (small-encoder, small-decoder) for stage 1 and VIM-Large for stage 2 (model architectures are listed in Table 2), as we ﬁnd it beneﬁcial to put more computation in stage 2 while keeping stage 1 transformers lightweight.
We also present a model size ablation study and comparison with VQGAN in the Appendix.
Models are trained with a global training batch size of 1024 for a total of 450 , 000 training steps.
We use Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 .
9 and β 2 = 0 .
96 with the learning rate linearly warming up to a peak constant value of 4 .
5 × 10 − 4 over the ﬁrst 5000 steps and then exponentially decaying to 1 × 10 − 5 starting from 80 , 000 steps.
To save memory, we use a factorized version of Adam, Adafactor (Shazeer &
Stern, 2018), with the ﬁrst moment quantized into Int8 and factorized second moments.
No other techniques like mixed-precision training, model sharding, or gradient compression is used.
All models are trained with an input image resolution 256 × 256 on CloudTPUv4.
[Table]
Our main results on unconditional image synthesis on CelebA-HQ and FFHQ are summarized in
Table 5.
Without top- k and top- p (nucleus) sampling heuristics, we achieve FID of 7.0 on CelebAHQ and 5.3 on FFHQ, signiﬁcantly better than VQGAN (Esser et al., 2021).
Table 6 shows classconditioned image synthesis models on ImageNet, following Section 4.1.
Based on ViT-VQGANSS, we achieve IS of 175.1 and FID of 4.17, improving over the IS of 70.6 and FID of 17.04 with vanilla VQGAN.
When applied with classiﬁer-based rejection sampling, the ViT-VQGAN based model further achieves best FID of 3.04 and best Inception Score of 321.7.
Qualitative results are sampled and shown in Figure 2 (see the Appendix for more).
5. U NSUPERVISED L EARNING
After the generative pretraining to autoregressively model the density of ViT-VQGAN quantized image tokens, we evaluate the learned representation under the common linear protocol on ImageNet classiﬁcation.
We follow the same training hyper-parameters as the unconditional image synthesis models on ImageNet, and use ViT-VQGAN-SS image quantizer for better training throughput.
As discussed in Section 3.3, the ViT-VQGAN-SS image quantizer is trained without perceptual loss for unsupervised learning (perceptual loss is based on a supervised VGG network trained on ImageNet, which may make comparison unfair).
We apply an average pooling over the token features at a speciﬁc transformer block l from totally L blocks.
Similar to ﬁndings reported in iGPT (Chen et al., 2020a), the representations from the middle transformer blocks has better linear-probe accuracy (more study can be found in Appendix).
Speciﬁcally, we use the Transformer block of index(36 blocks in total) for VIM-Large and index 10 (24 blocks in total) for VIM-Base (architecture conﬁgurations are listed in Table 2).
[Table]
Noroozi & Favaro, 2016b)
[Table]
[Table]
(Donahue & Simonyan, 2019b)
[Table]
Table 7 shows the comparisons among different approaches divided into two categories: discriminative pretraining methods to distinguish among cropped or augmented image patches; and generative pretraining methods to generate image pixels or patches.
The linear-probe accuracy of our proposed
VIM with ViT-VQGAN are superior to other generative pretraining approaches like iGPT, and competitive with discriminative pretraining methods like BYOL (Grill et al., 2020) and DINO (Caron et al., 2021).
Speciﬁcally, ImageNet-pretrained VIM-L signiﬁcantly improves over iGPT-L, increasing linear-probe accuracy from 60.3% to 73.2% for a similar model size.
VIM-L also outperforms iGPT-XL, which is larger and trained with extra web image data.
Moreover, we also compare different stage 1 quantizers including CNN-based VQGAN and pretrained DALL-E dVAE (trained with extra web-scale image data) 2 in Table 7; these results are all worse than ViT-VQGAN quantizer, suggesting the importance of the multiple changes deﬁned in Section 3.
E THICS
Tasks involving generation raise a number of issues that should be considered, such as possible biases in underlying models and data—especially with respect to capabilities for people with different demographic backgrounds.
The three datasets used in this paper–ImageNet, CelebA-HQ, and
FFHQ–are all widely used in the literature, but it is worthwhile highlighting their unique natures and recent scholarship around them.
The FFHQ dataset 3 contains 70,000 images collected from Flickr, all of which have licenses appropriate for sharing, and the data maintainers provide means for individuals to opt-out of inclusion in the dataset.
FFHQ was speciﬁcally collected to cover a broad range of demographics with respect to faces of people.
This is conﬁrmed in our models’ generated examples, which cover a broad range of perceived ages, genders and ethnicities.
Nevertheless, Balakrishnan et al.
(2020) provide an extensive analysis of multiple forms of bias in datasets (including CelebA-HQ and FFHQ) and algorithms for face generation; not only do they ﬁnd imbalances in skin tone in FFHQ, but also correlations between multiple attributes such as skin tone and hair length.
Based on this and other factors such as privacy and copyright, they argue that synthetically-created face datasets, for which multiple attributes can be controlled, is an important direction of investment and general inquiry.
The CelebA-HQ dataset covers celebrities, which brings a consequent bias toward images of attractive people who are mostly in age range of twenty to forty years old.
Esser et al.
(2020) discusses these biases in details, and they furthermore project images from the FFHQ dataset onto CelebAHQ: the main effect of which is to produce images of younger people with features conforming more to norms of celebrities popular in the United States of America.
Our model’s generations appear to have a similar bias as derived from training on CelebA-HQ.
Neverethless, they do show broad coverage of different perceived genders and ethnicities, but with age skewed to the 20-40 year old range.
ImageNet is, of course, quite pervasive in computer vision.
In this paper, we learn to generate images given ImageNet class labels; these labels mostly concern animals, plants and things.
People are sometimes generated when conditioning on classes such as sunglasses since the training data images contain people wearing sunglasses, but the generated images contain few depictions of people overall.
Nevertheless, it is important to recognize that ImageNet itself was created with biases in terms of image selection and label annotation as a result of its process of creation (Denton et al., 2021).
Given this, results present on ImageNet cover a signiﬁcant, but nonetheless biased, sample of the kinds of scenes and objects one might encounter across the entire world.
There are also potential problematic aspects of image generation models, as demonstrated with biases found in the PULSE model (Menon et al., 2020)
(see Section 6) and in model correlations with human biases found in social psychology (Steed & Caliskan, 2021), as well as with possible uses of such models to create fake media (Westerlund, 2019).
A L INEAR - PROBE ON I MAGE N ET
We show linear-probe accuracy from different layers in a pretrained VIM-Base Transformer model in Figure 3.
Similar to iGPT (Chen et al., 2020a), we also ﬁnd the last few layers may not be the best layers for discriminative features, as the generative pretraining objective is to recover the original image tokens.
The linearprobe accuracy increases quickly from the ﬁrst transformer output, reaches its peak at middle layers and ﬁnally decreases for the last few blocks.
Interestingly, we ﬁnd for both VIMBase and VIM-Large, the middle transformer block has the near-best result.
This observation connects the transformer model to an encoderdecoder model where the encoder encodes image tokens into high-level semantic features and the decoder takes feature information to generate output image tokens.
We leave for future study regrading the interpretability of pretrained VIM models.
B M ODEL S IZES OF C LASS - CONDITIONED I MAGE N ET S YNTHESIS
We also present results of different sizes of Stage 2 Transformers for class-conditioned image synthesis and compare with VQGAN (Esser et al., 2021)
4 summarized in Table
[Table]
C I MPLEMENTATION D ETAILS OF F ACTORIZED C ODEBOOK
As we introduced in Section 3.
[Table]
we use a linear projection to reduce the encoded embedding to a low-dimensional variable space for code lookup.
A detailed illustration is shown in Figure
[Table]
D M ORE S AMPLES ON C LASS - CONDITIONED I MAGE N ET S YNTHESIS
House Finch
Great Grey Owl
Terrapin
Komodo Dragon
Night Snake
Peacock
Irish Terrier
Norfolk Terrier
Norwich Terrier
Yorkshire Terrier
Wood
[Figure]
Crock Pot
Lumbermill
Scale
Strawberry
Grand Piano
Guenon Monkey
Anemone Fish
Jay
Photocopier