U NCONDITIONAL I MAGE -T EXT P AIR G ENERATION WITH M ULTIMODAL C ROSS Q UANTIZER
A BSTRACT
Though deep generative models have gained a lot of attention, most of the existing works are designed for the unimodal generation task.
In this paper, we explore a new method for unconditional image-text pair generation.
We propose
MXQ-VAE, a vector quantization method for multimodal image-text representation.
MXQ-VAE accepts a paired image and text as input, and learns a joint quantized representation space, so that the image-text pair can be converted to a sequence of unified indices.
Then we can use autoregressive generative models to model the joint image-text representation, and even perform unconditional image-text pair generation.
Extensive experimental results demonstrate that our approach effectively generates semantically consistent image-text pair and also enhances meaningful alignment between image and text.
I NTRODUCTION
Deep generative models are an active research area, but the main research focus has been unimodal generation, either unconditional (GAN (Goodfellow et al., 2014), VAE (Kingma & Welling, 2013),
PixelCNN (Van Oord et al., 2016), GPT (Brown et al., 2020)) or conditional (Oscar (Li et al., 2020),
DALL-E (Ramesh et al., 2021)).
As of yet, unconditional multimodal generation has not been much explored.
One previous work (Shimanuki, 2019) proposes generating image and text simultaneously with a GAN-based approach.
However, the core idea was to treat text as an image, where the model generates a pair of images, one for the picture and one for the text, thus this process must undergo the OCR process (Smith, 2007).
To overcome this limitation, we propose Multimodal Cross Quantizer VAE (MXQ-VAE) for unconditional image-text pair generation and enhancing the semantic consistency of the generated image-text pair.
More specifically, we adopt a vector quantization method (Oord et al., 2017) and
Transformer encoder (Vaswani et al., 2017) that learns a joint quantized representation space.
Thus, we can convert the image-text pair to a sequence of unified indices, and train Autoregressive Transformer (Brown et al., 2020) over this unified indices sequence, allowing for generating semantically consistent image-text pair simultaneously without any conditional input.
In order to show the effectiveness of our approach, we design several baselines and construct the corrupted data, denoted as the Degree dataset, by gradually adjusting the degree of the alignment between image and text.
The experimental results demonstrate that our approach achieves consistent results on the Degree dataset, while baselines fail, and also can generate semantically consistent image-text pair over baselines.
The contributions of our work can be summarized as follows:
• For the first time, we propose MXQ-VAE, a vector quantization method that learns a joint quantized representation space for unconditional image-text pair generation • The quantitative and qualitative results show the effectiveness of our approach over baselines with considerable semantic consistency of the generated image-text pair on textaugmented MNIST, denoted as Caption MNIST and Oxford Flower-102 (Nilsback & Zisserman, 2008).
• Additionally, The experimental result on the Degree dataset demonstrates that our approach learns meaningful alignment between image and text compared to non-MXQ module model.
RELATED WORKS
Image and Text Generation Most image and text generation studies focus on unimodal generation.
(Li et al., 2020; Ramesh et al., 2021) propose image-to-text and text-to-image generation respectively.
(Kingma & Welling, 2013; Goodfellow et al., 2014; Van Oord et al., 2016) propose unconditional image generation task and (Brown et al., 2020) generates text.
(Shimanuki, 2019), on the other hand, proposes unconditional image-text generation, but it generates text of an image and must undergo the OCR (Smith, 2007).
Vector Quantized Variational Autoencoder VQ-VAE (Oord et al., 2017) is first proposed to represent images into the discrete representation.
(Razavi et al., 2019) enhances the VQ-VAE by using multi-scale hierarchical encoder.
(Ramesh et al., 2021) utilizes gumbel-softmax relaxation to better optimize the VQ-VAE.
The Transformer Family The Transformer (Vaswani et al., 2017) is an encoder-decoder architecture.
(Chen et al., 2020b; Zhou et al., 2020; Li et al., 2020) utilize Transformer encoder to jointly learn imagetext representation.
(Brown et al., 2020) is an autoregressive language model based on Transformer decoder.
Recently, by factorizing image and video into the discrete representation, (Brown et al., 2020) also can model the continuous data and has shown remarkable results (Chen et al., 2020a; Yan et al., 2021).
METHOD
Figure 1 illustrates our approach.
Following DALL-E, we adopt a two-stage approach.
-VAE decodes the sampled sequences to image-text pair
3. S TAGE 1: L EARN A JOINT QUANTIZED REPRESENTATION SPACE
Inspired by VQ-VAE, we design MXQ-VAE with CNN-based encoders, decoders (2D CNN for image and 1D CNN for text), linear layers, Transformer encoder and codebook C ∈ R K × D , where K and D are the size and dimension of C .
• Given an input image
I ∈
R H
× W × 3 and text T ∈ R w × d , each encoder E
i and E t encodes the input into a set of feature maps z i ∈ R H ′
× W ′ × D and z t ∈ R w ′ × D , where H, W and w are height, width of image and length of text, respectively, and are downsampled to H ′ , W ′ and w ′ by a factor of 8 , d is a dimension of the word embedding.
• To learn a joint quantized representation space, we design a MXQ module with Transformer encoder and codebook C .
Transformer encoder takes the concatenation of z i and z t , then outputs joint representation.
For more deeper representation, we drop the the part of the sequence z t from the joint representation then the remaining part z x ∈ R L × D , where L = H ′ × W ′ , is discretized onto a joint quantized representation space by searching a nearest neighbor in codebook C and produces a sequence of unified indices ˆ z ∈ R L × D .
• Each D i and D t in the Figure 1 consists of a linear layer and a decoder (i.e. reversed encoder).
To ensure the decoder takes desired size as input, the linear layer is applied to the spatial dimension (i.e. L ) of the ˆ z then the decoder reconstructs the original input from the reduced ˆ z .
3. S TAGE 2: U NCONDITIONAL IMAGE - TEXT PAIR GENERATION WITH A UNIFIED SEQUENCE
We adopt Autoregressive Transformer to model the joint representation with the sequence of unified indices.
During training, MXQ-VAE encodes image and text to the unified sequence and then Autoregressive Transformer is trained to predict the next index of the unified sequence.
At inference,
MXQ-VAE decodes the sequence sampled from the model to image-text pair
EXPERIMENTS 4. D ATASET
1) Caption MNIST Based on (Shin et al., 2021), we build 600k image-text pairs, each image-text pair contains several color, digit and position (Figure 2 in Appendix).
we have 4 colors (white, red, green and blue), 10 digits (0 to 9) and 5 positions (center, top left, top right, bottom left and bottom right).
According to the filled quadrant, we called each pair as Single and Quad1 to 4.
Single image-text pairs only have a color and digit, this colored digit is located in the center of the image .
2) Degree Dataset
We construct the Degree dataset based on Caption MNIST by gradually adjusting the degree of the alignment between image and text (Figure 2 in Appendix).
More specifically, we replace the color and digit in text to other random color and digit.
For instance, the Quad4 can have 5 degrees range from totally unpaired (Degree 0) to originally paired (Degree 4).
3) Oxford Flower-102 contains 81,890 images of 102 flower species, with 10 captions per image.
4. B ASELINES
All baselines also generate image-text pair without any conditional input, and there is only a difference in which modality comes first.
1) Only Sharing e .
At Stage 1, this baseline only shares the codebook e without any module to jointly combine both image and text.
At Stage 2, depending on which modality comes first, we refer to it as I&T VQ or T&I VQ. 2)
Separately Quantizing.
At Stage 1, two distinct models quantize each modality separately.
For image, it follows original VQ-VAE.
There are two versions for text, (1) it also follows VQ-VAE; (2) since text is originally discrete data, we directly use the word embedding.
At Stage 2, depending on which modality comes first, in case (1), we refer to it as I T VQ or T I VQ, and in case (2), we refer to it as I(VQ) T(Embd) or T(Embd) I(VQ).
3) MXQ-VAE (w/o text compression).
We change the E t and D t of MXQ-VAE to the Transformer encoder.
In this model, the text encoder outputs the same number of embeddings as the input.
4. R ESULTS
For the details of evaluation metrics, please refer to Appendix A.1.
We first study the effect of the MXQ module with respect to the alignment between image and text .
We evaluate all models on the Degree dataset.
Table 1 and Table 4, 5, 6, and 7 in Appendix show the results.
We observe that Only Sharing e and MXQ-VAE (w/o text compression) can not fully capture the alignment between image and text.
In fact, MXQ-VAE (w/o text compression) totally fails to learn the relationship between image and text.
Our approach, on the other hand, shows the best approximation, close to 1.0, 0.75, 0.5, 0.25 and 0.0 on Quad4.
Also, we examine the advantage of the sequence drop ( i.e. dropping z t ) that brings in considerable improvement.
We measure the semantic consistency of the generated image-text pairs on Caption MNIST.
Table 2 shows the performance of all models.
We can observe that MXQ-VAE outperforms all baselines on every quadrant.
Also note that all baselines are vulnerable to which modality is given first to generate the image-text pair.
All models given text first significantly degrade up to 27% (in
I T VQ and T I VQ) on average compared to the model given image first.
We assume that this is due to the fact that quantized image is longer than text and image often contains more complex information than text.
Our MXQ-VAE, on the other hand, alleviates this issue with a unified sequence.
The generated image-text pairs (Figure 3 in Appendix) show the high fidelity image and paired text.
[Table] [Table]
We also assess our approach on Oxford Flower-102.
Table 3 shows the brief result, refer to Tablein Appendix for the whole result.
Compared with our baseline model Joint GAN (Shimanuki, 2019),
MXQ-VAE outperforms in all scores, indicating its capability to generate semantically consistent flower image-text pairs.
This result shows that our approach can extend to more general domain in the future.
Figure 4 and 5 in Appendix show the generated image-text pairs by MXQ-VAE and Joint GAN respectively.
As we can see, MXQ-VAE generates more realistic image and high quality text.
In summary, quantitative and qualitative results demonstrate that our approach enhances the meaningful alignment between image and text, and it also encourages to generate semantically consistent image-text pairs.
[Table]
CONCLUSION
In this study, we first propose MXQ-VAE, a vector quantization method for unconditional imagetext pair generation that quantize the paired image and text to a joint quantized representation space.
Extensive experimental results demonstrate the effectiveness of our approach.
In future work, we plan on extending our approach to a large-scale complex dataset.
A CKNOWLEDGEMENTS
This work was supported by Institute of Information & Communications Technology
Planning & Evaluation (IITP) grant (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)) and National Research Foundation of Korea (NRF) grant (NRF2020H1D3A2A03100945) funded by the Korea government (MSIT), and the KAIST-NAVER Hyper-Creative AI Center
A A PPENDIX A. E VALUATION M ETRICS
• Semantic parser
Following (Shin et al., 2021), we evaluate the semantic consistency between the generated image and text on Caption MNIST.
We extract a set of position, color, and digit with the rule-based semantic parser of the generated text.
With a color and digit classifier trained with the image of Caption MNIST and achieved 100% and 99.5% accuracy each, we predict the color and digit of the part of the generated image corresponding to the position that exist in the generated text.
Then, we measure whether both predicted color and digit math or not at that position.
• Alignment accuracy It measures the alignment of image and text using Degree dataset.
For instance, in Quad1, the Degree 1 input text is ’the green 0 is on the upper right.’
and the Degree 0 input text is ’the white 1 is on the upper right.’.
Input image always depicts the original text, Degree 1 in this case.
If the reconstructed text is ’the green 0 is on the upper right.’
for both Degree 1 and 0 input text, the alignment accuracy will be 1 and 0 for each degree.
With the wellalignment space, the model should figure out the corrupted part and does not reconstruct that part as it is.
According to this, in Quad3, the desired accuracy would be 1, 0.66, 0.33 and 0 for each degree.
• Sentence similarity To evaluate the semantic consistency of the generated image-text pairs on Oxford Flower-102 (Nilsback & Zisserman, 2008), we measure a sentence similarity between the generated text and the pooled original text that has the same class.
Specifically, we first train a shallow CNN on the original image to classify the 82 categories and achieves 99% accuracy.
We predict the class of the generated image with this classifier and pool all the original text that has the same class.
The pre-trained BERT (Devlin et al., 2018) takes the generated text paired with that image and the pooled original text respectively and outputs each vector by mean pooling operation.
We calculate the cosine similarity between the vector of the generated text and the others and report a Top-1, 5 and 10 averaged score.
To validate
this metric, we conduct on
original
Oxford Flower-102 and achieve 0.999, 0.963 and 0.962 average similarity score respectively.
A. I MPLEMENTATIONS
• Stage 1 We use 2D-, 1D-CNN encoder and decoder for images and text respectively.
For the Caption MNIST, the embedding latent space e is 256 × 128 , the input size is 64 × 64 × 3 image and 64 × 128 word embedding, and then each input is downsampled to 8 × 8 × 128 and 8 × 128 .
For Oxford Flower-102, the embedding latent space e is 512 ×
the input size is 224 × 224 ×image and 80 × 128 word embedding.
Each input is
downsampled
to
28 × 28 ×
64 and
×
We split text into word based on white space.
For MXQ-VAE (w/o text compression) model, we use Transformer encoder with 2 layers, 2 attention heads and 128 embedding dimension.
We train model with 1 NVIDIA RTX A6000 with 800 and 496 batch size for Caption MNIST and Oxford
Flower-102 respectively.
Following (Oord et al., 2017), Our model optimizes using the following objectives:
[Equation]
The loss term consists of reconstruction losses for image and text, a codebook loss, and a commitment loss.
sg refers to a stop-gradient.
The commitment loss is weighted by a hyperparameter β
and we set β to 0.25 and
.5 for Caption
MNIST and Oxford Flower-102 respectively
• Stage 2 We adopt (Brown et al., 2020) for an autoregressive generative model with 8 layers,attention heads and 512 embedding dimension.
The model optimizes negative log-likelihood loss.
We
train model
with 1 NVIDIA RTX A6000 with 800 and 40 batch size for Caption MNIST
and
Oxford Flower-102.
In all our experiments, we use AdamW (Loshchilov & Hutter, 2017) with β 1 = 0 .
9 , β 2 = 0 .
99 , and the cosine learning rate scheduler (Loshchilov & Hutter, 2016) from 5 e − 4 to 0 multiplied by the batch size.
[Table]
[Table] [Table]
[Table] [Table]