Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation (2024)

Abstract: Recent advances in image tokenizers, such as VQ-VAE, have enabled text-to-image generation using auto-regressive methods, similar to language modeling. However, these methods have yet to leverage pre-trained language models, despite their adaptability to various downstream tasks. In this work, we explore this gap by adapting a pre-trained language model for auto-regressive text-to-image generation, and find that pre-trained language models offer limited help. We provide a two-fold explanation by analyzing tokens from each modality. First, we demonstrate that image tokens possess significantly different semantics compared to text tokens, rendering pre-trained language models no more effective in modeling them than randomly initialized ones. Second, the text tokens in the image-text datasets are too simple compared to normal language model pre-training data, which causes the catastrophic degradation of language models’ capability.

## introduction recent works in text-to-image generation primarily employ two kinds of methods: diffusion models @xcite @xcite and auto-regressive models @xcite . the latter is facilitated by "image tokenizers", such as vq-vae @xcite and vq-gan @xcite , which transform an image into a sequence of discrete tokens, similar to text tokens (figure 2 left). consequently, image and text tokens can be jointly modeled using auto-regressive algorithms like the transformer @xcite (figure 2 right). figure 1: auto-regressive and diffusion based models achieve similar performances on text-to-image generation. however, while all the diffusion models leverage pre-trained language models, all the auto-regressive models do not. a comparison based on zero-shot fid @xcite on the coco dataset @xcite can be found in figure 1 . while these two approaches achieve similar performance, it is intriguing that diffusion-based models consistently utilize pre-trained text encoders, whereas their auto-regressive counterparts generally do not. for instance, imagen @xcite (diffusion-based) reports that employing a stronger pre-trained text encoder, specifically t5 @xcite , yields substantial improvements to using clip @xcite . furthermore, they observe that scaling up the t5 text encoder leads to more pronounced improvements @xcite ) and converted to a grid of discrete tokens, and it can be well-reconstructed with these image tokens. (right) as images are converted to tokens similar to text tokens, we can enable language models to generate images by adapting its embedding layer and output layer. in this work, we explore the potential of pretrained llms for auto-regressive text-to-image generation. to enable the model to process both text and image tokens, we expand the size of the embedding and output layers by incorporating an image vocabulary from the image tokenizer. we initialize these added weights either randomly or using a novel contrastive alignment (elaborated later in section 3.2), while the remaining weights are directly copied from the original models. subsequently, we fine-tune the model on image-caption datasets, as depicted in figure 2 right. surprisingly, the results show that pre-trained language models achieve the same loss and image generation quality as the model that is entirely randomly initialized and trained from scratch (figure 3 ). furthermore, we observe a catastrophic deterioration in the model's text capabilities, such as world knowledge or in-context learning, after only minimal steps of fine-tuning ( to understand this phenomenon, we break down the cross-entropy loss on image and text tokens, and find that 1) the loss on image tokens is the same between the pre-trained and randomly initialized model, and 2) the loss on text tokens of the pre-trained model is significantly lower at the beginning compared to the randomly initialized models, but the gap soon disappears after training (figure 4 ). the first finding of the loss on the image tokens is particularly interesting. we hypothesize that im-age tokens obtained from image tokenizers might either lack semantics or possess significantly different semantics compared to text tokens, which renders language pre-training not transferable to the image modeling task. to verify this hypothesis, we conduct unconditional image generation experiments by training the model on image tokens only. our results show that 1) the pre-trained model achieves the same loss as the randomly initialized model, and 2) freezing any part of the pretrained model results in a loss degradation (figure 6 ). these indicate that optimal weights for language and image modeling are fundamentally different, making language pre-training not transferable to image modeling. in summary, we share our experimental findings about pre-trained language models do not help autoregressive text-to-image generation, and offer an explanation: 1) the intrinsic differences between image and text tokens make language pre-training ineffective for the image token modeling, and 2) the disproportionate ratio between image and text tokens (usually 30:1 for image-caption datasets) minimizes the impact of loss on text tokens and leads to catastrophic forgetting. ## image tokens are drastically different from text tokens why there is no difference between the loss of pretrained and randomly initialized models on the image tokens? we hypothesize image tokens are significantly different from text tokens, for example, they lack semantics or have drastically different semantics compared to text tokens, which makes the pre-trained language model not transferable to we also compare the full fine-tuning with electively finetuning components of the pre-trained models (shown in parenthesis). ema 0.95 is applied to the plot. ## conclusion this study highlights the difficulty of naively adapting a text-only language model to handle multimodal contents, such as texts and images. given the challenge of the disparities between image tokens and text tokens, a valuable avenue for future experiments is to employ tokenizers that align semantically with text tokens, such as seed @xcite or spae @xcite .