Original: T HE PLUG AND PLAY OF LANGUAGE MODELS FOR TEXT - TO - IMAGE GENERATION
Revised: T HE P LUG AND P LAY OF L ANGUAGE M ODELS FOR T EXT - TO - IMAGE G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Text-to-image (T2I) models enable controllable image generation through userprovided captions.
Revised: Text-to-image (T2I) models enable controllable image generation through userprovided captions.

Original: A text encoder is typically used to map captions to a latent space, and it has been shown to be critical for model’s performance.
Revised: A text encoder is typically used to map captions to a latent space, and it has been shown to be critical for model’s performance.

Original: However, replacing or upgrading the text encoder in a T2I model is challenging due to the tight bond between the current encoder and the image decoder.
Revised: However, replacing or upgrading the text encoder in a T2I model is challenging due to the tight bond between the current encoder and the image decoder.

Original: It requires training the model from scratch, which can be prohibitively expensive.
Revised: It requires training the model from scratch, which can be prohibitively expensive.

Original: To address this problem, we introduce a more efficient approach to align a pre-trained language model with the latent space of an existing T2I model.
Revised: To address this problem, we introduce a more efficient approach to align a pre-trained language model with the latent space of an existing T2I model.

Original: We propose a Model Translation
Revised: We propose a Model Translation

Original: Network (MTN) and a new training objective to align the representation spaces of the two text encoders using only a corpus of unlabeled text.
Revised: Network (MTN) and a new training objective to align the representation spaces of the two text encoders using only a corpus of unlabeled text.

Original: We empirically find that MTN can be trained efficiently and can boost the performance of existing T2I models by upgrading their text encoder.
Revised: We empirically find that MTN can be trained efficiently and can boost the performance of existing T2I models by upgrading their text encoder.

Original: Moreover, we find that MTN can align multilingual language models such as XLM-Roberta, thus allowing existing T2I models to generate high-quality images from captions beyond English.
Revised: Moreover, we find that MTN can align multilingual language models such as XLM-Roberta, thus allowing existing T2I models to generate high-quality images from captions beyond English.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text-to-image (T2I) generative models have made great progress in the last few years thanks to algorithmic advances and the availability of large-scale paired training datasets (Ramesh et al., 2022; Yu et al., 2022a; Saharia et al., 2022; Rombach et al., 2022).
Revised: Text-to-image (T2I) generative models have made great progress in the last few years thanks to algorithmic advances and the availability of large-scale paired training datasets (Ramesh et al., 2022; Yu et al., 2022a; Saharia et al., 2022; Rombach et al., 2022).

Original: Diffusion-based T2I generative models in particular have achieved remarkable results in terms of image quality (Ho & Salimans, 2022; Nichol et al., 2021).
Revised: Diffusion-based T2I generative models in particular have achieved remarkable results in terms of image quality (Ho & Salimans, 2022; Nichol et al., 2021).

Original: Despite these strong results, controllable generation for these methods is still challenging: generated images are often not faithful to the captions, compositional capabilities are lacking, and prompt engineering is often required to achieve the desired results (Parsons, 2022).
Revised: Despite these strong results, controllable generation for these methods is still challenging: generated images are often not faithful to the captions, compositional capabilities are lacking, and prompt engineering is often required to achieve the desired results (Parsons, 2022).

Original: Moreover, most large-scale models have only been trained on English captions, greatly limiting their use across the world.
Revised: Moreover, most large-scale models have only been trained on English captions, greatly limiting their use across the world.

Original: To improve T2I models, recent research suggests that the ability of their text encoders to understand and represent text is critical and is a bottleneck for their image generation performance (Saharia et al., 2022; Croitoru et al., 2022).
Revised: To improve T2I models, recent research suggests that the ability of their text encoders to understand and represent text is critical and is a bottleneck for their image generation performance (Saharia et al., 2022; Croitoru et al., 2022).

Original: Text encoders in existing T2I models are often trained only on short image captions, and their performance on complex prompts can be largely constrained by the quality of the features extracted by the text encoder (
Revised: Text encoders in existing T2I models are often trained only on short image captions, and their performance on complex prompts can be largely constrained by the quality of the features extracted by the text encoder (

Original: Rombach et al., 2022).
Revised: Rombach et al., 2022).

Original: However, upgrading the text encoder for an existing T2I model is challenging because the representation spaces of the text encoder and image generator are tightly coupled (Rombach et al., 2022; Ramesh et al., 2022).
Revised: However, upgrading the text encoder for an existing T2I model is challenging because the representation spaces of the text encoder and image generator are tightly coupled (Rombach et al., 2022; Ramesh et al., 2022).

Original: Training only the text encoder on a more complex and representative text corpus would break this alignment, hindering the final image generation performance.
Revised: Training only the text encoder on a more complex and representative text corpus would break this alignment, hindering the final image generation performance.

Original: Training the entire T2I model from scratch (perhaps with higher quality image-caption pairs) would be prohibitively expensive (Edwards, 2022).
Revised: Training the entire T2I model from scratch (perhaps with higher quality image-caption pairs) would be prohibitively expensive (Edwards, 2022).

Original: To solve this problem, we propose a method that can efficiently align off-the-shelf pre-trained language models with image encoders of existing diffusion-based T2I models.
Revised: To solve this problem, we propose a method that can efficiently align off-the-shelf pre-trained language models with image encoders of existing diffusion-based T2I models.

Original: With this method, the existing text encoder of T2I models can be replaced with a more powerful language model or even one for a language other than English as illustrated in Fig.
Revised: With this method, the existing text encoder of T2I models can be replaced with a more powerful language model or even one for a language other than English as illustrated in Fig.

Original: Crucially, the representation alignment between text and image encoders is maintained without retraining from scratch.
Revised: Crucially, the representation alignment between text and image encoders is maintained without retraining from scratch.

Original: We refer to our proposed method as Model Translation Network (MTN).
Revised: We refer to our proposed method as Model Translation Network (MTN).

Original: MTN follows an encoder-decoder structure.
Revised: MTN follows an encoder-decoder structure.

Original: Specifically, given a trained T2I model and a pre-trained language model that we want to replace, the
Revised: Specifically, given a trained T2I model and a pre-trained language model that we want to replace, the

Original: [Figure]
Revised: [Figure]

Original: XLM
Revised: XLM

Original: [Figure]
Revised: [Figure]

Original: T5-L
Revised: T5-L

Original: Generated Image <prompt>
Revised: Generated Image <prompt>

Original: Upgrading Aligning
Revised: Upgrading Aligning

Original: Text Conditioning
Revised: Text Conditioning

Original: A dv a n ce d L a ngu a g e M od e l s // //
Revised: A dv a n ce d L a ngu a g e M od e l s // //

Original: New Text Encoder
Revised: New Text Encoder

Original: Old Text Encoder
Revised: Old Text Encoder

Original: Obtaining
Revised: Obtaining

Original: MTN Encoder encoder of MNT firstly aligns the representation space of the pre-trained language model with that of the T2I model’s image generator, by minimizing both element-wise and global-wise discrepancy.
Revised: MTN Encoder encoder of MNT firstly aligns the representation space of the pre-trained language model with that of the T2I model’s image generator, by minimizing both element-wise and global-wise discrepancy.

Original: Then, the decoder of MNT takes the aligned text representations as inputs, and maps them back to the original representation space of the pre-trained language model by minimizing the reconstruction loss.
Revised: Then, the decoder of MNT takes the aligned text representations as inputs, and maps them back to the original representation space of the pre-trained language model by minimizing the reconstruction loss.

Original: The reason why we need this decoder during training is that some recent research reveals that training to align existing models inevitably decreases feature discriminability (Chen et al., 2019;
Revised: The reason why we need this decoder during training is that some recent research reveals that training to align existing models inevitably decreases feature discriminability (Chen et al., 2019;

Original: Cui et al., 2020).
Revised: Cui et al., 2020).

Original: Therefore we need to preserve rich semantics captured by the pre-trained language model during the alignment training, ensuring that a decoder can recover the original representation space of the pre-trained language model from the aligned one.
Revised: Therefore we need to preserve rich semantics captured by the pre-trained language model during the alignment training, ensuring that a decoder can recover the original representation space of the pre-trained language model from the aligned one.

Original: The entire training of MTN requires only a corpus of unlabeled text.
Revised: The entire training of MTN requires only a corpus of unlabeled text.

Original: At inference time, only the encoder of MTN is applied on top of the pre-trained language model for representation alignment.
Revised: At inference time, only the encoder of MTN is applied on top of the pre-trained language model for representation alignment.

Original: To verify the effectiveness of the proposed framework, we have applied a stronger language model, i.e , T5-3B (Raffel et al., 2020), to upgrade existing text encoders of the Latent Diffusion
Revised: To verify the effectiveness of the proposed framework, we have applied a stronger language model, i.e , T5-3B (Raffel et al., 2020), to upgrade existing text encoders of the Latent Diffusion

Original: Model (Rombach et al., 2022).
Revised: Model (Rombach et al., 2022).

Original: The competitive improvements in the FID score and user study ranking reveal the benefits of our model over the baselines.
Revised: The competitive improvements in the FID score and user study ranking reveal the benefits of our model over the baselines.

Original: Furthermore, our model has the potential to bring new functionalities into existing T2I models, such as multilingual text-to-image generation.
Revised: Furthermore, our model has the potential to bring new functionalities into existing T2I models, such as multilingual text-to-image generation.

Original: We empirically find that MTN can align a multilingual language model such as XLMRoberta-L (Conneau et al., 2019) with existing T2I models.
Revised: We empirically find that MTN can align a multilingual language model such as XLMRoberta-L (Conneau et al., 2019) with existing T2I models.

Original: It enables the existing image generator to understand text beyond English, such as French and Chinese, and to generate high-quality images accordingly.
Revised: It enables the existing image generator to understand text beyond English, such as French and Chinese, and to generate high-quality images accordingly.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • To the best of our knowledge, this is the first work to consider the problem of efficiently aligning a pre-trained language model with a pre-trained T2I diffusion model.
Revised: • To the best of our knowledge, this is the first work to consider the problem of efficiently aligning a pre-trained language model with a pre-trained T2I diffusion model.

Original: • Extensive experiments on text-to-image generation benchmarks demonstrate the superiority of our model over the baseline LDM method on both image quality and language controllability.
Revised: • Extensive experiments on text-to-image generation benchmarks demonstrate the superiority of our model over the baseline LDM method on both image quality and language controllability.

Original: • Our framework also enables text-to-image generation beyond English prompts without the need of multilingual image-text pairs for retraining.
Revised: • Our framework also enables text-to-image generation beyond English prompts without the need of multilingual image-text pairs for retraining.

Original:  R ELATED W ORKS 2. T EXT - TO -I MAGE G ENERATION
Revised:  R ELATED W ORKS 2. T EXT - TO -I MAGE G ENERATION

Original: Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) is one of the major frameworks in text-to-image generation.
Revised: Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) is one of the major frameworks in text-to-image generation.

Original: The method in (Reed et al., 2016) was an early stage approach to use a GAN to train a text-to-image generation network.
Revised: The method in (Reed et al., 2016) was an early stage approach to use a GAN to train a text-to-image generation network.

Original: Since then, other GAN based methods, e.g ,
Revised: Since then, other GAN based methods, e.g ,

Original: Stack-GAN (Zhang et al., 2017), Attn-GAN (Xu et al., 2018) and SD-GAN (Yin et al., 2019), have obtained promising results.
Revised: Stack-GAN (Zhang et al., 2017), Attn-GAN (Xu et al., 2018) and SD-GAN (Yin et al., 2019), have obtained promising results.

Original: In addition, recent works showed more improvements on the generation quality.
Revised: In addition, recent works showed more improvements on the generation quality.

Original: DM-GAN (Zhu et al., 2019) improved text-to-image performance by introducing a dynamic memory component.
Revised: DM-GAN (Zhu et al., 2019) improved text-to-image performance by introducing a dynamic memory component.

Original: DF-GAN (Tao et al., 2022) designed a fusion module to fuse text and image features.
Revised: DF-GAN (Tao et al., 2022) designed a fusion module to fuse text and image features.

Original: LAFITE
Revised: LAFITE

Original: (Zhou et al., 2021) took advantage of CLIP’s model to construct pseudo imagetext pairs, and proposed a GAN model to learn text-image pairs.
Revised: (Zhou et al., 2021) took advantage of CLIP’s model to construct pseudo imagetext pairs, and proposed a GAN model to learn text-image pairs.

Original: Auto-regressive transformer is another major framework in text-to-image generation.
Revised: Auto-regressive transformer is another major framework in text-to-image generation.

Original: DALL-E (Ramesh et al., 2021) and CogView (Ding et al., 2021) adopted an autoregressive transformer (Vaswani et al., 2017) to train the correspondences between visual tokens and text tokens.
Revised: DALL-E (Ramesh et al., 2021) and CogView (Ding et al., 2021) adopted an autoregressive transformer (Vaswani et al., 2017) to train the correspondences between visual tokens and text tokens.

Original: Parti (Yu et al., 2022a) used a powerful visual encoder, ViT-VQGAN (Yu et al., 2022b), to improve results upon DALL-E.
Revised: Parti (Yu et al., 2022a) used a powerful visual encoder, ViT-VQGAN (Yu et al., 2022b), to improve results upon DALL-E.

Original: The method in (Gafni et al., 2022) is similar to CogView and DALL-E, while introducing additional controlling elements to improve the tokenization process.
Revised: The method in (Gafni et al., 2022) is similar to CogView and DALL-E, while introducing additional controlling elements to improve the tokenization process.

Original: N ¨UWA (Wu et al., 2022b) and NUWA-infinity (Wu et al., 2022a) trained autoregressive visual synthesis model to support both text-to-image and text-to-video generation tasks.
Revised: N ¨UWA (Wu et al., 2022b) and NUWA-infinity (Wu et al., 2022a) trained autoregressive visual synthesis model to support both text-to-image and text-to-video generation tasks.

Original: Concurrently, diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song et al., 2020) became a main research focus.
Revised: Concurrently, diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song et al., 2020) became a main research focus.

Original: In such methods, noise is added to an image, and a score network is trained to denoise and recover the input image.
Revised: In such methods, noise is added to an image, and a score network is trained to denoise and recover the input image.

Original: GLIDE (Nichol et al., 2021) performed guided inference with and without a classifier network to generate high-fidelity images.
Revised: GLIDE (Nichol et al., 2021) performed guided inference with and without a classifier network to generate high-fidelity images.

Original: DALLE 2 (Ramesh et al., 2022) and Imagen (Saharia et al., 2022) set new state-of-the-art results in the text-to-image generation.
Revised: DALLE 2 (Ramesh et al., 2022) and Imagen (Saharia et al., 2022) set new state-of-the-art results in the text-to-image generation.

Original: In DALL-E 2, a prior to produce CLIP image embeddings conditioned on text was learned, and a diffusion model was used to decode the image embeddings to an image.
Revised: In DALL-E 2, a prior to produce CLIP image embeddings conditioned on text was learned, and a diffusion model was used to decode the image embeddings to an image.

Original: In Imagen, a frozen large-scale text encoder T5-XXL (Raffel et al., 2020) was adopted to generate embeddings, and a cascade of conditional diffusion models was used to map these embeddings to images of increasing resolutions.
Revised: In Imagen, a frozen large-scale text encoder T5-XXL (Raffel et al., 2020) was adopted to generate embeddings, and a cascade of conditional diffusion models was used to map these embeddings to images of increasing resolutions.

Original: Latent diffusion model and stable diffusion model (Rombach et al., 2022) are state-of-the-art methods that apply the diffusion model on the latent embedding space as in (Sinha et al., 2021; Vahdat et al., 2021).
Revised: Latent diffusion model and stable diffusion model (Rombach et al., 2022) are state-of-the-art methods that apply the diffusion model on the latent embedding space as in (Sinha et al., 2021; Vahdat et al., 2021).

Original: These methods are computationally friendly while achieving impressive results.
Revised: These methods are computationally friendly while achieving impressive results.

Original: 2. K NOWLEDGE D ISTILLATION (KD)
Revised: 2. K NOWLEDGE D ISTILLATION AND C ROSS - LINGUAL A LIGNMENT

Original: Knowledge distillation is a technique to guide the training of a student network with a reference teacher network (Bucilu ˇ a et al., 2006).
Revised: Knowledge distillation is a technique to guide the training of a student network with a reference teacher network (Bucilu ˇ a et al., 2006).

Original: (Hinton et al., 2014) applied probabilities of the teacher as a target for the student to learn.
Revised: (Hinton et al., 2014) applied probabilities of the teacher as a target for the student to learn.

Original: Since then, many methods around the idea of “what knowledge to be distilled” have been proposed.
Revised: Since then, many methods around the idea of “what knowledge to be distilled” have been proposed.

Original: Popular definitions include distance of features (Romero et al., 2015), map attention (Zagoruyko & Komodakis, 2017), distribution-wise distillation (Passalis & Tefas, 2018) and inter-sample distance structure (Tung & Mori, 2019), etc.
Revised: Popular definitions include distance of features (Romero et al., 2015), map attention (Zagoruyko & Komodakis, 2017), distribution-wise distillation (Passalis & Tefas, 2018) and inter-sample distance structure (Tung & Mori, 2019), etc.

Original: Despite the great success, they mainly focus on the classification task with discrete outputs and pay less attention to representation protection.
Revised: Despite the great success, they mainly focus on the classification task with discrete outputs and pay less attention to representation protection.

Original: Translating dense and high-dimensional features between large-scale language models requires new solutions.
Revised: Translating dense and high-dimensional features between large-scale language models requires new solutions.

Revised: Cross-lingual alignment methods attempt to align the representation across different languages based on a multilingual text model (Kulshreshtha et al.

Original: P RELIMINARY 3. L ATENT D IFFUSION M ODEL
Revised: ; Cao et al., 2020). (Kulshreshtha et al.) has comprehensively explored the different alignment methods, including rotation and finetuning alignment. (Cao et al., 2020) has modeled the contextual token-wise embeddings to aggregate richer information for alignment. Our paper considers another problem in aligning different text models with the same or parallel data.

Revised: This is essentially different from cross-lingual alignment, with more complicated and challenging mismatches such as feature dimensions and variant token lengths.

Original: The Latent Diffusion Model (LDM) (Rombach et al., 2022) and its extension, i.e , Stable Diffusion, which are variants of Denoising Diffusion Probabilistic Model (DDPM)
Revised: The Latent Diffusion Model (LDM) (Rombach et al., 2022) and its extension, i.e , Stable Diffusion, which are variants of Denoising Diffusion Probabilistic Model (DDPM)

Original: (Ho et al., 2020) family, are selected as our baseline models due to their excellent balance in efficiency and visual quality.
Revised: (Ho et al., 2020) family, are selected as our baseline models due to their excellent balance in efficiency and visual quality.

Original: In general, LDM is composed of two stages.
Revised: In general, LDM is composed of two stages.

Original: Firstly, there is an auto-encoder (AE) pre-trained by enormous images with the regularization in either KL-divergence or vector quantization (Van
Revised: Firstly, there is an auto-encoder (AE) pre-trained by enormous images with the regularization in either KL-divergence or vector quantization (Van

Original: Den Oord et al., 2017; Agustsson et al., 2017).
Revised: Den Oord et al., 2017; Agustsson et al., 2017).

Original: An encoder network, i.e , E , is applied for latent feature extraction as z = E ( x ) , which can be mapped back to image space by a decoder network.
Revised: An encoder network, i.e , E , is applied for latent feature extraction as z = E ( x ) , which can be mapped back to image space by a decoder network.

Original: The input images x and reconstructed images ˆ x are almost identical x
Revised: The input images x and reconstructed images ˆ x are almost identical x

Original: ≈ ˆ x .
Revised: ≈ ˆ x .

Original: Secondly, LDM trains a diffusion model in the latent space (Sinha et al., 2021; Vahdat et al., 2021).
Revised: Secondly, LDM trains a diffusion model in the latent space (Sinha et al., 2021; Vahdat et al., 2021).

Original: It follows the standard DDPM
Revised: It follows the standard DDPM

Original: (Ho et al., 2020) with denoising loss and uses U-net (Ronneberger et al., 2015)
Revised: (Ho et al., 2020) with denoising loss and uses U-net (Ronneberger et al., 2015)

Original: as the image decoder as in (Song & Ermon, 2019).
Revised: as the image decoder as in (Song & Ermon, 2019).

Original: To enable generative controllability, 
Revised: To enable generative controllability, [Figure]

Revised: 𝓛 𝒎𝒔𝒆 𝒙 𝒔𝒓𝒄 𝓛 𝒂𝒅𝒗 𝓛 𝒓 𝒆 𝒄 𝒙 𝒕𝒂𝒓 𝒙" 𝒔𝒓𝒄 𝒙 𝒂𝒍𝒊𝒈𝒏

Revised: Fixed New Text Encoder 𝑭 𝒕 𝑭 𝒔 𝑴 𝑵 𝑫 𝒙 𝒕𝒂𝒓 𝒙 𝒕𝒂𝒓 𝒙 𝒕𝒂𝒓 𝒙 𝒔𝒓𝒄 𝒙 𝒂𝒍𝒊𝒈𝒏 𝒙# 𝒔𝒓𝒄

Original:  LDM has applied multiple conditioning signals ( y ) such as text, mask, or layout, encoded aside and injected into the U-net, with the help of cross-attention layers.
Revised: M o d e l T r a n s l a t i o n N e t w o r k ( M T N ) [Figure] LDM has applied multiple conditioning signals ( y ) such as text, mask, or layout, encoded aside and injected into the U-net, with the help of cross-attention layers.

Original: This can be formulated as:
Revised: This can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where t represents the time step, z t is the noise corrupted latent tensor at time step t , and z 

Revised: where t represents the time step, z t is the noise corrupted latent tensor at time step t , and z

Original: = E ( x ) .
Revised: = E ( x ) .

Original: ε is the unscaled Gaussian noise, c
Revised: ε is the unscaled Gaussian noise, c

Original: ϕ is the conditioning network parameterized by ϕ and ε θ is the Unet-like denoising network ( a.k.a , image decoder).The parameters of both conditioning and [Figure]
Revised: ϕ is the conditioning network parameterized by ϕ and ε θ is the Unet-like denoising network ( a.k.a , image decoder).

Original: 𝓛 𝒎𝒔𝒆 𝒙 𝒔𝒓𝒄 𝓛 𝒂𝒅𝒗 𝓛 𝒓 𝒆 𝒄 𝒙 𝒕𝒂𝒓 𝒙" 𝒔𝒓𝒄 𝒙 𝒂𝒍𝒊𝒈𝒏

Original: Fixed New Text Encoder 𝑭 𝒕 𝑭 𝒔 𝑴 𝑵 𝑫 𝒙 𝒕𝒂𝒓 𝒙 𝒕𝒂𝒓 𝒙 𝒕𝒂𝒓 𝒙 𝒔𝒓𝒄 𝒙 𝒂𝒍𝒊𝒈𝒏 𝒙# 𝒔𝒓𝒄

Original: M o d e l T r a n s l a t i o n N e t w o r k ( M T N ) [Figure] denoising networks, i.e , θ, ϕ , are trained by the LDM loss.
Revised: The parameters of both conditioning and denoising networks, i.e , θ, ϕ , are trained by the LDM loss.

Original: During inference, clean images can be generated via classifier-free guidance (Ho & Salimans, 2022) as:
Revised: During inference, clean images can be generated via classifier-free guidance (Ho & Salimans, 2022) as:

Original: [Equation]
Revised: [Equation]

Original: where s is the guidance weight to balance text controllability and image fidelity.
Revised: where s is the guidance weight to balance text controllability and image fidelity.

Original: The basic T2I LDM model is trained on Laion-400M (Schuhmann et al., 2021) dataset.
Revised: The basic T2I LDM model is trained on Laion-400M (Schuhmann et al., 2021) dataset.

Original: The Stable
Revised: The Stable

Original: Diffusion Model (SDM) was trained with more epochs and training data, i.e , Laion2B-en and Laionhigh-resolution.
Revised: Diffusion Model (SDM) was trained with more epochs and training data, i.e , Laion2B-en and Laionhigh-resolution.

Original: 3. L ANGUAGE M ODELS FOR T2I
Revised: 3. L ANGUAGE M ODELS FOR T2I

Original: LDM uses Bert (Devlin et al., 2018) as the text encoder jointly trained with the image decoder using the DDPM loss.
Revised: LDM uses Bert (Devlin et al., 2018) as the text encoder jointly trained with the image decoder using the DDPM loss.

Original: SDM uses a pre-trained CLIP (Radford et al., 2021) text encoder frozen during training.
Revised: SDM uses a pre-trained CLIP (Radford et al., 2021) text encoder frozen during training.

Original: It has been found that increasing the size of the text encoder is very helpful for performance (Saharia et al., 2022).
Revised: It has been found that increasing the size of the text encoder is very helpful for performance (Saharia et al., 2022).

Original: In order to upgrade existing models, our goal is to be able to efficienlty plug in more advanced language models such as T5-3B (Raffel et al., 2020) with more parameters or XLM-Roberta (Devlin et al., 2018) for multilingual T2I.
Revised: In order to upgrade existing models, our goal is to be able to efficienlty plug in more advanced language models such as T5-3B (Raffel et al., 2020) with more parameters or XLM-Roberta (Devlin et al., 2018) for multilingual T2I.

Original:  M ODEL T RANSLATION N ETWORK (MTN)
Revised:  M ODEL T RANSLATION N ETWORK (MTN)

Original: The text encoder is one of the key components of T2I models, as generation requires precise and finegrained text embedding for guidance.
Revised: The text encoder is one of the key components of T2I models, as generation requires precise and finegrained text embedding for guidance.

Original: The overall performance can be greatly boosted by increasing the size of the text encoder (Saharia et al., 2022), however, upgrading an existing text model to a more powerful one is challenging.
Revised: The overall performance can be greatly boosted by increasing the size of the text encoder (Saharia et al., 2022), however, upgrading an existing text model to a more powerful one is challenging.

Original: Because existing T2I models are not modular, directly replacing (or augmenting with) another pre-trained language model does not work as seen in Fig.
Revised: Because existing T2I models are not modular, directly replacing (or augmenting with) another pre-trained language model does not work as seen in Fig.

Original: 9 (a).
Revised: 9 (a).

Original: The key technical challenge is the mis-alignment between the new language model and the old image decoder.
Revised: The key technical challenge is the mis-alignment between the new language model and the old image decoder.

Original: Moreover, joint finetuning also falls short due to catastrophic forgetting occuring when updating well trained parameters.
Revised: Moreover, joint finetuning also falls short due to catastrophic forgetting occuring when updating well trained parameters.

Original: Considering these two different text models as source and target domains, we apply a neural network to learn to align.
Revised: Considering these two different text models as source and target domains, we apply a neural network to learn to align.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, to efficiently upgrade the text encoder and simultaneously protect existing knowledge, this paper has presented a general framework called Model Translation
Revised: 2, to efficiently upgrade the text encoder and simultaneously protect existing knowledge, this paper has presented a general framework called Model Translation

Original: Network (MTN).
Revised: Network (MTN).

Original: MTN works as an additional module to bridge the new language model and the old image decoder as seen in Fig.
Revised: MTN works as an additional module to bridge the new language model and the old image decoder as seen in Fig.

Original: 4. O BJECTIVES
Revised: 4. O BJECTIVES

Revised: Given a set of text corpus X = { x ij | i = 1 , ..., M, j = 1 , ..., N } with M as length of tokens and N as total quantity, the source language model F s and a target text encoder F t have mapped the raw text data into embeddings denoted as S =

Revised: { s ij | i = 1 , ..., M s , j = 1 , ..., N } where s ij =

Original: The MTN is an autoencoder whose encoder M ( · , Θ M  ) learns tomap the new text features from a source language model F s to align with the current target text encoder F t .
Revised: F s ( x ij ) ∈ R d s and T = { t ij | i = 1 , .., M t , j = 1 , ..., N } where t ij = F t ( x ij ) ∈ R d t . Due to the different models, there is severe distribution mismatch between the source and target features

Original: This translation (  i.e , 
Revised: p ( s ) ̸ = p ( t ) . The MTN is an autoencoder whose encoder M ( · , Θ M ) learns to map the new text features from a source language model, i.e , s ∈ S , to align with the current target text encoder, i.e , t ∈ T , where p ( M ( s, Θ M )) ≈ p ( t ) .

Original:  F s Θ M −→ F t ) enables the image generator to understand the new text features coming from the new language model without finetuning.
Revised: Therefore, this translation ( i.e , F s Θ M −→ F t ) enables the image generator to understand the new text features coming from the new language model without finetuning.

Original: To achieve this aim, we consider it as a domain adaptation (Ganin et al., 2016; Qin et al., 2019;
Revised: To achieve this aim, we consider it as a domain adaptation (Ganin et al., 2016; Qin et al., 2019;

Original: Long et al., 2018) problem and apply both element-wise and distribution-wise alignment losses, which includes the minimization of the mean square error (MSE) loss, i.e , L mse , and the adversarial loss, i.e , L adv , measured over a discriminator network, i.e , D ( · , Θ D ) : 
Revised: Long et al., 2018) problem and apply both element-wise and distribution-wise alignment losses, which includes the minimization of the mean square error (MSE) loss, i.e , L mse , and the adversarial loss, i.e , L adv , measured over a discriminator network, i.e , D ( · , Θ D ) : [Equation] mse [Equation]

Revised: [Equation]

Revised: s

Original: [Equation]
Revised: [Equation]

Original: where X s and X tdenote the source and target inputs respectively and they can be the same prompts in English ( e.g , LDM-Bert → T5-3B (Raffel et al., 2020)) or in bilingual but parallel content for multilingual T2I generation.
Revised: X s [ || F t ( [Equation] [Equation] s

Revised: [Equation]

Revised: [Equation]

Revised: L adv (

Revised: [Equation]

Revised: where X s and X t denote the source and target inputs respectively and they can be the same prompts in English ( e.g , LDM-Bert → T5-3B (Raffel et al., 2020)) or in bilingual but parallel content for multilingual T2I generation.

Original: Some insightful investigations in transfer learning reveal that the cross-domain alignment will inevitably decrease feature discrimination (Chen et al., 2019; Cui et al., 2020).
Revised: Some insightful investigations in transfer learning reveal that the cross-domain alignment will inevitably decrease feature discrimination (Chen et al., 2019; Cui et al., 2020).

Original: To project the rich semantics necessary for model upgrading, we further apply a decoder network, i.e , N ( · , Θ N ) , for feature reconstruction:
Revised: To project the rich semantics necessary for model upgrading, we further apply a decoder network, i.e , N ( · , Θ N ) , for feature reconstruction:

Original: [Equation]
Revised: [Equation]

Original: MTN is trained in an end-to-end manner and the details of which can be referred to in Algorithmand Sec.
Revised: MTN is trained in an end-to-end manner and the details of which can be referred to in Algorithmand Sec.

Original: (Ma et al., 2022) introduced that a universal learning engine should seek a compressive closed-loop transcription with good property in structure-preserving.
Revised: (Ma et al., 2022) introduced that a universal learning engine should seek a compressive closed-loop transcription with good property in structure-preserving.

Original: Our cross-model mapping also requires a similar functionality.
Revised: Our cross-model mapping also requires a similar functionality.

Original: MTN desires stability after a loop whose input and output should keep almost equivalent, i.e , x ≈
Revised: MTN desires stability after a loop whose input and output should keep almost equivalent, i.e , x ≈

Original: ˆ x ≈ ˆˆ
Revised: ˆ x ≈ ˆˆ

Original: x
Revised: x

Original: ≈ ... , due to the constraint of reconstruction loss.
Revised: ≈ ... , due to the constraint of reconstruction loss.

Original: Therefore, inputting the reconstructed data ˆ x to MTN is expected to be consistent with the previous loop.
Revised: Therefore, inputting the reconstructed data ˆ x to MTN is expected to be consistent with the previous loop.

Original: This is helpful for accelerating MTN training.
Revised: This is helpful for accelerating MTN training.

Original: 4. A RCHITECTURE OF M ODEL T RANSLATION N ETWORK
Revised: 4. A RCHITECTURE OF M ODEL T RANSLATION N ETWORK

Original: [Figure]
Revised: [Figure]

Original: The input data to alignment can be represented as the sequence of tokens: X i, ∗ = { x i, 1 , x i, 2 , ..., x i,S } where X i, ∗ ∈ X and x i,j ∈ R C , with sequence length S and token dim C .
Revised: The input data to alignment can be represented as the sequence of tokens: X i, ∗ = { x i, 1 , x i, 2 , ..., x i,S } where X i, ∗ ∈ X and x i,j ∈ R C , with sequence length S and token dim C .

Original: Inspired by the methods of image super-resolution (Lim et al., 2017;
Revised: Inspired by the methods of image super-resolution (Lim et al., 2017;

Original: Ahn et al., 2018) to densely regress target data under different input-output dimensions, the translator network stacks three sub-nets in sequence, as seen in Fig.
Revised: Ahn et al., 2018) to densely regress target data under different input-output dimensions, the translator network stacks three sub-nets in sequence, as seen in Fig.

Original: The head net is only used for simple feature transformation.
Revised: The head net is only used for simple feature transformation.

Original: The body net has many residual modules for fine-grained representation learning.
Revised: The body net has many residual modules for fine-grained representation learning.

Original: The tail network is appended and is mainly employed for dimension conversion to match the target tensors.
Revised: The tail network is appended and is mainly employed for dimension conversion to match the target tensors.

Original: Every residual module is implemented based on the MLP-based token mixer (Tolstikhin et al., 2021; Ma et al., 2021) with high efficiency and strong ability in representation learning.
Revised: Every residual module is implemented based on the MLP-based token mixer (Tolstikhin et al., 2021; Ma et al., 2021) with high efficiency and strong ability in representation learning.

Original: Within the MLPmixer block, it consists of a Token Mixer and a Sequence Mixer for orthometric purposes.
Revised: Within the MLPmixer block, it consists of a Token Mixer and a Sequence Mixer for orthometric purposes.

Original: The
Revised: The

Original: Token Mixer learns the representation of each token with shared MLP, and the Sequence Mixer is designed for learning the same channel in sequential tokens:
Revised: Token Mixer learns the representation of each token with shared MLP, and the Sequence Mixer is designed for learning the same channel in sequential tokens:

Original: [Equation]
Revised: [Equation]

Original: where X · , · is the feature, LN γ,β denotes layer normalization parameterized by γ and β , and σ indicates the non-linear operator such as GELU (Hendrycks & Gimpel, 2016).
Revised: where X · , · is the feature, LN γ,β denotes layer normalization parameterized by γ and β , and σ indicates the non-linear operator such as GELU (Hendrycks & Gimpel, 2016).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we comprehensively evaluate our proposed framework for the monolingual and multilingual text-to-image generation upon upgrading the text encoder of pre-trained models.
Revised: In this section, we comprehensively evaluate our proposed framework for the monolingual and multilingual text-to-image generation upon upgrading the text encoder of pre-trained models.

Original: More details can be referred to in Supplementary Materials .
Revised: More details can be referred to in Appendix .

Original: 5. E XPERIMENTS S ETUP
Revised: 5. E XPERIMENTS S ETUP

Original: Upon the shoulders of giants, this paper applies the Latent-Diffusion Model (LDM) and Stable
Revised: Upon the shoulders of giants, this paper applies the Latent-Diffusion Model (LDM) and Stable

Original: Diffusion Model (SDM) (Rombach et al., 2022) as the baselines with a similar model but different text encoders and scalabilities.
Revised: Diffusion Model (SDM) (Rombach et al., 2022) as the baselines with a similar model but different text encoders and scalabilities.

Original: LDM uses a Bert (Devlin et al., 2018) as the text encoder that is jointly updated with the image decoder based on DDPM loss, while SDM uses a frozen pretrained CLIP text encoder during training.
Revised: LDM uses a Bert (Devlin et al., 2018) as the text encoder that is jointly updated with the image decoder based on DDPM loss, while SDM uses a frozen pretrained CLIP text encoder during training.

Original: The LDM is designed for the images in 256 × 256 and
Revised: The LDM is designed for the images in 256 × 256 and

Original: SDM targets 512 × 512 images.
Revised: SDM targets 512 × 512 images.

Original: Considering our limitations in data and computation resources, the experiments on monolingual T2I are based on LDM with a lower cost of text encoder upgrading than that of SDM.
Revised: Considering our limitations in data and computation resources, the experiments on monolingual T2I are based on LDM with a lower cost of text encoder upgrading than that of SDM.

Original: We implement the multilingual T2I generation upon the SDM.
Revised: We implement the multilingual T2I generation upon the SDM.

Original: Implementation.
Revised: Implementation.

Original: The MTN is implemented as the Fig.
Revised: The MTN is implemented as the Fig.

Original: 3 with an encoder M and a decoder N in symmetrical architecture consisting ∼ 51M parameters.
Revised: 3 with an encoder M and a decoder N in symmetrical architecture consisting ∼ 51M parameters.

Original: For simplicity in denotation, we refer MTN encoder as MTN in the following experiments, whose body net consists of 5 residual modules as default.
Revised: For simplicity in denotation, we refer MTN encoder as MTN in the following experiments, whose body net consists of 5 residual modules as default.

Original: We take the AdamW (Loshchilov & Hutter, 2017) as the optimizer based on PyTorch (Paszke et al., 2019).
Revised: We take the AdamW (Loshchilov & Hutter, 2017) as the optimizer based on PyTorch (Paszke et al., 2019).

Original: The learning rate is assigned as 1 × 10 − 4 for MTN training and 1 × 10 − 6 for LDM-T5MTN finetuning.
Revised: The learning rate is assigned as 1 × 10 − 4 for MTN training and 1 × 10 − 6 for LDM-T5MTN finetuning.

Original: The MTN model is trained on Nvidia-A100-40GB, requiring 1 to 5 GPU days.
Revised: The MTN model is trained on Nvidia-A100-40GB, requiring 1 to 5 GPU days.

Original: [Figure]
Revised: [Figure]

Original: Language Models.
Revised: Language Models.

Original: We apply the T53B (Raffel et al., 2020) to upgrade the text encoder of LDM Bert.
Revised: We apply the T53B (Raffel et al., 2020) to upgrade the text encoder of LDM Bert.

Original: T5-3B has 6x more parameters than LDM Bert and is trained by an enormous text corpus beyond the image captions.
Revised: T5-3B has 6x more parameters than LDM Bert and is trained by an enormous text corpus beyond the image captions.

Original: Furthermore, to enable multilingual T2I generation, we choose the XLM-Roberta-L (Devlin et al., 2018) as the new text encoder to let SDM image decoder understand languages beyond English.
Revised: Furthermore, to enable multilingual T2I generation, we choose the XLM-Roberta-L (Devlin et al., 2018) as the new text encoder to let SDM image decoder understand languages beyond English.

Original: Datasets.
Revised: Datasets.

Original: We have collected the text data for MTN training.
Revised: We have collected the text data for MTN training.

Original: In monolingual T2I, only English texts are required, and we extract 18M captions from Laion-400M as training corpus, and more data will be more helpful.
Revised: In monolingual T2I, only English texts are required, and we extract 18M captions from Laion-400M as training corpus, and more data will be more helpful.

Original: Multilingual MTNs require parallel texts in different languages.
Revised: Multilingual MTNs require parallel texts in different languages.

Original: These data are collected by Wikimarix (Schwenk et al., 2019) and Googletrans with ∼ 2M sentences for each.
Revised: These data are collected by Wikimarix (Schwenk et al., 2019) and Googletrans with ∼ 2M sentences for each.

Original: Ablations.
Revised: Ablations.

Original: T5+MTN refers to the LDM model with the stacking of T5 and MTN as the text encoder where the LDM image decoder is unchanged.
Revised: T5+MTN refers to the LDM model with the stacking of T5 and MTN as the text encoder where the LDM image decoder is unchanged.

Original: T5+FT represents finetuning of the LDM image decoder with DDPM loss based on text conditions from a fixed T5.
Revised: T5+FT represents finetuning of the LDM image decoder with DDPM loss based on text conditions from a fixed T5.

Original: T5+MTN+FT is the final version, including both MTN and joint finetuning with the image decoder.
Revised: T5+MTN+FT is the final version, including both MTN and joint finetuning with the image decoder.

Original: The finetuning takes ∼ 100 GPU days of 116M image-text pairs filtered from Laion-400M by BLIP score (Li et al., 2022). 
Revised: The finetuning takes ∼ 100 GPU days of 116M image-text pairs filtered from Laion-400M by BLIP score (Li et al., 2022). 5. M ONOLINGUAL T EXT - TO -I MAGE G ENERATION

Revised: Monolingual T2I generation focuses on the comparison with LDM given the English text prompts.

Revised: We have decomposed our pipeline into two stages: alignment and finetuning.

Revised: MTN helps to align

Original: L D M
Revised: L D M

Original: O r i g i n a l
Revised: O r i g i n a l

Original: T 5 + M
Revised: T 5 + M

Original: T N
Revised: T N

Original: T 5 + F
Revised: T 5 + F

Original: T
Revised: T

Original: T 5 + M
Revised: T 5 + M

Original: T N + F
Revised: T N + F

Original: T a cube made of brick a painting by grant wood of an astronaut couple, american gothic style a sign that says text to image darth vader playing-with raccoon in mars during sunset a person walking in a ﬁeld of red ﬂowers with a yellow dress a person walking in a ﬁeld of yellow ﬂowers with a red dress a horse getting wet 5. M ONOLINGUAL T EXT - TO -I MAGE G ENERATION
Revised: T a cube made of brick a painting by grant wood of an astronaut couple, american gothic style a sign that says text to image darth vader playing-with raccoon in mars during sunset a person walking in a ﬁeld of red ﬂowers with a yellow dress a person walking in a ﬁeld of yellow ﬂowers with a red dress a horse getting wet the new language model whose potential can be better exploited by further finetuning of image text pairs.

Original: Monolingual T2I generation focuses on the comparison with LDM given the English text prompts.

Original: We have decomposed our pipeline into two stages: alignment and finetuning.

Original: MTN helps to align the new language model whose potential can be better exploited by further finetuning of image text pairs.

Original: This section verifies the benefits of our vanilla and finetuned models in both quantitative and qualitative evaluations.
Revised: This section verifies the benefits of our vanilla and finetuned models in both quantitative and qualitative evaluations.

Original: 5.2. Q UANTITATIVE E VALUATION
Revised: 5.2. Q UANTITATIVE E VALUATION

Original: Tab. 1 reported the zero-shot FID score (Heusel et al., 2017) on COCO dataset (Lin et al., 2014) based on the pyorch-fid (Seitzer, 2020) package.
Revised: Tab. 1 reported the zero-shot FID score (Heusel et al., 2017) on COCO dataset (Lin et al., 2014) based on the pyorch-fid (Seitzer, 2020) package.

Original: Our re-implementation score of LDM is slightly higher than the paper’s one (Rombach et al., 2022), which may because of different implementations.
Revised: Our re-implementation score of LDM is slightly higher than the paper’s one (Rombach et al., 2022), which may because of different implementations.

Original: The table shows the minor drop of T5+MTN from LDM since our current MTN model still leaves some incorrect alignments, which could be fixed by either joint finetuning or collecting more text corpus to train MTN.
Revised: The table shows the minor drop of T5+MTN from LDM since our current MTN model still leaves some incorrect alignments, which could be fixed by either joint finetuning or collecting more text corpus to train MTN.

Original: With the ∼ 100 GPU-days finetuning of a subset of the Laion dataset ( i.e , 116M image-text pairs), the overall performance has been considerably improved, even outperforming the
Revised: With the ∼ 100 GPU-days finetuning of a subset of the Laion dataset ( i.e , 116M image-text pairs), the overall performance has been considerably improved, even outperforming the

Original: LDM by certain margins.
Revised: LDM by certain margins.

Original: [Table]
Revised: [Table]

Original: We have also conducted a user study to compare the image quality and text controllability in Fig.
Revised: We have also conducted a user study to compare the image quality and text controllability in Fig.

Original: 4 by Amazon Turk (Buhrmester et al., 2011).
Revised: 4 by Amazon Turk (Buhrmester et al., 2011).

Original: There are ∼ 1K prompts collected from Drawbench (Saharia et al., 2022) and Winoground (Thrush et al., 2022), which comprehensively cover various scenarios and numerous difficult cases.
Revised: There are ∼ 1K prompts collected from Drawbench (Saharia et al., 2022) and Winoground (Thrush et al., 2022), which comprehensively cover various scenarios and numerous difficult cases.

Original: We report the percentages of users’ votes for the three methods in Fig.
Revised: We report the percentages of users’ votes for the three methods in Fig.

Original: The study of image quality is designed to evaluate the resolution, cleanness, and smoothness of object boundaries
Revised: The study of image quality is designed to evaluate the resolution, cleanness, and smoothness of object boundaries, where our method shows fragile superiority.

Original: [Table]

Original: where our method shows fragile superiority. In the user study of controllability, ours outperformed the LDM by 3%, which is a non-trivial improvement. 
Revised: In the user study of controllability, ours outperformed the LDM by 3%, which is a non-trivial improvement. Moreover, the p-value 2 is computed as 0. with three repeats of user study that is statistically significant according to the criteria of < 0.05.

Original: [Figure]
Revised: [Figure]

Original: 5.2. V ISUAL C OMPARISON
Revised: 5.2. V ISUAL C OMPARISON

Original: In Fig.
Revised: In Fig.

Original: we have exhibited some generated images of four different methods.
Revised: we have exhibited some generated images of four different methods.

Original: By comparing T5+FT and T5+MNT+FT with similar finetuning costs (100 GPU-days) we can conclude that the proposed
Revised: By comparing T5+FT and T5+MNT+FT with similar finetuning costs (100 GPU-days) we can conclude that the proposed

Original: MNT helps save much effort in plugging a new language model due to the high-quality images.
Revised: MNT helps save much effort in plugging a new language model due to the high-quality images.

Original: Moreover, the T5+MNT+FT shows stronger controllability in many cases compared with LDM, such as “a cube made of brick” and “a horse getting wet”, whereas the LDM’s results fail to display the concepts of “cube” and “wet”.
Revised: Moreover, the T5+MNT+FT shows stronger controllability in many cases compared with LDM, such as “a cube made of brick” and “a horse getting wet”, whereas the LDM’s results fail to display the concepts of “cube” and “wet”.

Original: 5. M ULTILINGUAL T EXT - TO -I MAGE G ENERATION
Revised: 5. M ULTILINGUAL T EXT - TO -I MAGE G ENERATION

Original: Our proposed framework is very general and generic to many language models, including the multilingual backbones.
Revised: Our proposed framework is very general and generic to many language models, including the multilingual backbones.

Original: We choose the recent SDM as our base model with the pre-trained clip-vitlarge-patch14 as the text encoder to be upgraded by the multilingual one, i.e , XLM-Roberta-L.
Revised: We choose the recent SDM as our base model with the pre-trained clip-vitlarge-patch14 as the text encoder to be upgraded by the multilingual one, i.e , XLM-Roberta-L.

Original: The
Revised: The

Original: MTN helps to align such two backbones with the parallel corpus as training data.
Revised: MTN helps to align such two backbones with the parallel corpus as training data.

Original: Thus, the old
Revised: Thus, the old

Original: SDM image decoder can successfully understand the text embeddings of other languages encoded by MTN and XLM-Roberta-L without any additional parameters updating except MTN.
Revised: SDM image decoder can successfully understand the text embeddings of other languages encoded by MTN and XLM-Roberta-L without any additional parameters updating except MTN.

Original: 5.3. V ISUAL C OMPARISON
Revised: 5.3. V ISUAL C OMPARISON

Original: Fig.
Revised: Fig.

Original: 6 shows the example of multilingual generation results with the prompt “portrait of a cat painted by Picasso” in many languages.
Revised: 6 shows the example of multilingual generation results with the prompt “portrait of a cat painted by Picasso” in many languages.

Original: Most of the results reveal the content of “portrait of a cat painted”.
Revised: Most of the results reveal the content of “portrait of a cat painted”.

Original: The “Picasso” is a challenging notion that rarely appears in our training data.
Revised: The “Picasso” is a challenging notion that rarely appears in our training data.

Original: Thus, more highquality parallel data will be greatly helpful.
Revised: Thus, more highquality parallel data will be greatly helpful.

Original: Compared with those finetuned over multilingual image text pairs, our solution is far more efficient and cheaper.
Revised: Compared with those finetuned over multilingual image text pairs, our solution is far more efficient and cheaper.

Original: 5.3. H YBRID M ULTILINGUAL G ENERATION
Revised: 5.3. H YBRID M ULTILINGUAL G ENERATION

Original: The hybrid multilingual generation is more challenging that both the original SDM and third-party translation toolbox cannot address.
Revised: The hybrid multilingual generation is more challenging that both the original SDM and third-party translation toolbox cannot address.

Original: Our proposed MTN has surprising capability in this problem.
Revised: Our proposed MTN has surprising capability in this problem.

Original: According to the results in Fig. 7, the multilingual backbone bonded with MTN gives the precise guide to the SDM decoder even with the prompts in three different languages and randomly mixed.
Revised: According to the results in Fig. 7, the multilingual backbone bonded with MTN gives the precise guide to the SDM decoder even with the prompts in three different languages and randomly mixed.

Original: 5. A LIGNMENT A NALYSIS
Revised: 5. A LIGNMENT A NALYSIS

Original: [Figure]
Revised: [Figure]

Original: ピカソが描いた猫
Revised: ピカソが描いた猫

Original: [Figure]
Revised: [Figure]

Original: sit
Revised: sit

Original: [Figure]
Revised: [Figure]

Original: a table
Revised: a table

Original: [Figure]
Revised: [Figure]

Original: 在吃
Revised: 在吃

Original: [Figure]
Revised: [Figure]

Original: “ ⼀ 辆 ⽩⾊的 , セダン , crash into a building”
Revised: “ ⼀ 辆 ⽩⾊的 , セダン , crash into a building”

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: To understand how MTN maps the crossmodel features, we have visualized its trajectory with the generative results in different iterations in Fig.
Revised: To understand how MTN maps the crossmodel features, we have visualized its trajectory with the generative results in different iterations in Fig.

Original: In the beginning, the old image decoder could not understand the new text embeddings at all.
Revised: In the beginning, the old image decoder could not understand the new text embeddings at all.

Original: Then, certain patterns such as “pencil drawing” appear after 300 iterations of training with a significant decrease in losses.
Revised: Then, certain patterns such as “pencil drawing” appear after 300 iterations of training with a significant decrease in losses.

Original: Such a gap can finally be sealed at the end where both L mse and L rec converged to small values.
Revised: Such a gap can finally be sealed at the end where both L mse and L rec converged to small values.

Original: 5. T-SNE F EATURE M APS WITH D IFFERENT MTN S IZES
Revised: 5. T-SNE F EATURE M APS WITH D IFFERENT MTN S IZES

Original: Fig.
Revised: Fig.

Original: 8 illustrates the feature maps of different text encoders.
Revised: 8 illustrates the feature maps of different text encoders.

Original: Source features are collected from
Revised: Source features are collected from

Original: LDM-Bert and target ones are achieved by T5-3B with the shared prompts from Drawbench and
Revised: LDM-Bert and target ones are achieved by T5-3B with the shared prompts from Drawbench and

Original: Winoground.
Revised: Winoground.

Original: RMs represents the number of the residual modules.
Revised: RMs represents the number of the residual modules.

Original: We can see that MTNs’ features are precisely aligned with those of the source text model.
Revised: We can see that MTNs’ features are precisely aligned with those of the source text model.

Original: There is no much difference between larger and smaller MTNs.
Revised: There is no much difference between larger and smaller MTNs.

Original: Considering efficiency, we choose the MTN-5RMs as the default model.
Revised: Considering efficiency, we choose the MTN-5RMs as the default model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Infusing the pre-trained language models into the existing T2I image generator is an existing direction toward a more powerful AI system.
Revised: Infusing the pre-trained language models into the existing T2I image generator is an existing direction toward a more powerful AI system.

Original: However, the current text encoder cannot be easily upgraded due to the tight match.
Revised: However, the current text encoder cannot be easily upgraded due to the tight match.

Original: This paper attempts to break such a strong constraint of the corresponding image-text models towards flexible modularization and efficient upgrading.
Revised: This paper attempts to break such a strong constraint of the corresponding image-text models towards flexible modularization and efficient upgrading.

Original: To address the severe misalignment, we have proposed the model translation network (MTN) with both objectives for cross-model alignment and originality protection.
Revised: To address the severe misalignment, we have proposed the model translation network (MTN) with both objectives for cross-model alignment and originality protection.

Original: Empirically, it is beneficial for the overall performance and enables versatile functionalities such as multilingual generation within a limited budget.
Revised: Empirically, it is beneficial for the overall performance and enables versatile functionalities such as multilingual generation within a limited budget.

Original: We wish this work to be inspirable to the community in large-scale AI system design.
Revised: We wish this work to be inspirable to the community in large-scale AI system design.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper strictly follows the ICLR ethical research standards and laws.
Revised: This paper strictly follows the ICLR ethical research standards and laws.

Original: All datasets we employed are publicly available, and all related publications and source codes are cited appropriately.
Revised: All datasets we employed are publicly available, and all related publications and source codes are cited appropriately.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We adhere to ICLR reproducibility standards and ensure the reproducibility of our work in multiple ways, including:
Revised: We adhere to ICLR reproducibility standards and ensure the reproducibility of our work in multiple ways, including:

Original: • Besides the codes, we will share the our processed data and model checkpoints to the public.
Revised: • Besides the codes, we will share the our processed data and model checkpoints to the public.

Original: • We clearly present the design of MTN and joint finetuning in Section 5.1 and Section 4.
Revised: • We clearly present the design of MTN and joint finetuning in Section 5.1 and Section 4.

Original: • Detailed framework and more experiments are presented in Supplementary Materials.
Revised: • Detailed framework and more experiments are presented in Supplementary Materials.

Original: A D ETAILS OF M ODEL T RANSLATION N ETWORK (MTN)
Revised: A D ETAILS OF M ODEL T RANSLATION N ETWORK (MTN)

Original: Model Translation Network (MTN) is the key contribution of this paper, which needs to address both cross-model alignment and feature protection.
Revised: Model Translation Network (MTN) is the key contribution of this paper, which needs to address both cross-model alignment and feature protection.

Original: In this section, we will provide more details of the MTN, including implementation/configuration, analysis of sizes, and other ablation studies.
Revised: In this section, we will provide more details of the MTN, including implementation/configuration, analysis of sizes, and other ablation studies.

Original: A. A RCHITECTURE AND C ONFIGURATION
Revised: A. A RCHITECTURE AND C ONFIGURATION

Original: We have introduced the basic architecture of MTN in Sec.
Revised: We have introduced the basic architecture of MTN in Sec.

Original: 4.2 in the main paper.
Revised: 4.2 in the main paper.

Original: MTN could have variants in different sizes whose configurations can be referred to in Tab. 2.
Revised: MTN could have variants in different sizes whose configurations can be referred to in Tab. 2.

Original: The MTN with three residual modules is the smaller one, with 34M parameters in its encoder.
Revised: The MTN with three residual modules is the smaller one, with 34M parameters in its encoder.

Original: The MTN-5RMs has 51M parameters.
Revised: The MTN-5RMs has 51M parameters.

Original: In general, larger-size models inevitably result in slower speed and more computation costs.
Revised: In general, larger-size models inevitably result in slower speed and more computation costs.

Original: Considering the sizes of the image decoder and text encoder as the basis, the MTN, working as an injected module, should not be larger than these major characters.
Revised: Considering the sizes of the image decoder and text encoder as the basis, the MTN, working as an injected module, should not be larger than these major characters.

Original: Otherwise, the inference speed will be significantly slower, and it may need huge computations in the finetuning stage.
Revised: Otherwise, the inference speed will be significantly slower, and it may need huge computations in the finetuning stage.

Original: We empirically found that assigning RM as five is a satisfactory choice.
Revised: We empirically found that assigning RM as five is a satisfactory choice.

Original: The MTN-3RMs seems too weak in representation learning.
Revised: The MTN-3RMs seems too weak in representation learning.

Original: Except for the MLP-mixer, the self-attention-based model may also be applied for implementing MTN.
Revised: Except for the MLP-mixer, the self-attention-based model may also be applied for implementing MTN.

Original: Exploring more suitable architectures of MTN is our future work.
Revised: Exploring more suitable architectures of MTN is our future work.

Original: [Table]
Revised: [Table]

Original: TokenMLP  
Revised: TokenMLP  

Revised: Linear , LN , σ

Original: [Table] Linear [Table]
Revised: Linear , LN , σ Linear , LN 

Original:    
Revised:   

Original: Linear , LN , 
Revised: Linear , LN , σ

Original: [Table]

Original: Linear , [Table]
Revised: Linear , LN , σ

Original: [Table] Linear , [Table] 
Revised: Linear , LN 

Original: 
Revised: 

Original: SequenceMLP  
Revised: SequenceMLP  

Original: Linear , [Table]
Revised: Linear , LN , σ

Original: [Table]

Original: Linear , LN , σ
Revised: Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original:   
Revised:   

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original: 
Revised: 

Original: [Table] Net
Revised: Body Net

Original: TokenMLP  
Revised: TokenMLP  

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN   × 
Revised: Linear , LN   × 

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original:  ×
Revised:  ×

Original: SequenceMLP  
Revised: SequenceMLP  

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN   × 
Revised: Linear , LN   × 

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original:  ×
Revised:  ×

Original: [Table] Net
Revised: Tail Net

Original: Token
Revised: Token

Original: MLP  
Revised: MLP  

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original:   
Revised:   

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original: 
Revised: 

Original: SequenceMLP  
Revised: SequenceMLP  

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original:   
Revised:   

Original: Linear , LN , σ Linear , LN , σ
Revised: Linear , LN , σ Linear , LN , σ

Original: Linear , LN 
Revised: Linear , LN 

Original: 
Revised: 

Original: A. E FFECTS OF MTN S IZES
Revised: A. E FFECTS OF MTN S IZES

Original: Fig.
Revised: Fig.

Original: 10 shows a visual comparison of some example prompts over three different methods.
Revised: 10 shows a visual comparison of some example prompts over three different methods.

Original: The
Revised: The

Original: LDM Ori refers to the original LDM model with the checkpoint shared in its repo.
Revised: LDM Ori refers to the original LDM model with the checkpoint shared in its repo.

Original: MTN-3RMs and
Revised: MTN-3RMs and

Original: MTN-5RMs are our models with 3 or 5 residual modules inside their body nets.
Revised: MTN-5RMs are our models with 3 or 5 residual modules inside their body nets.

Original: The text encoder is replaced as the T5-3B, and the image decoder is the same as LDM Ori.
Revised: The text encoder is replaced as the T5-3B, and the image decoder is the same as LDM Ori.

Original: Both MTN models are trained by the same text corpus, including 18M sentences.
Revised: Both MTN models are trained by the same text corpus, including 18M sentences.

Original: From the figure, we can notice that MTN3RMs fails on some complicated prompts such as “a virus monster is playing guitar, oil on canvas” and “there is a penguin with a dog head standing” where the MTN-5RMs performs well.
Revised: From the figure, we can notice that MTN3RMs fails on some complicated prompts such as “a virus monster is playing guitar, oil on canvas” and “there is a penguin with a dog head standing” where the MTN-5RMs performs well.

Original: Therefore, we can conclude that deep MTN is necessary for precise alignment.
Revised: Therefore, we can conclude that deep MTN is necessary for precise alignment.

Original: a blue car a virus monster is playing guitar, oil on canvas there is a penguin with a dog head standing an astronaut riding a horse as a pencil drawing
Revised: a blue car a virus monster is playing guitar, oil on canvas there is a penguin with a dog head standing an astronaut riding a horse as a pencil drawing

Original: LDM Ori
Revised: LDM Ori

Original: MTN-3RMs
Revised: MTN-3RMs

Original: MTN-5RMs
Revised: MTN-5RMs

Original: A. A BLATION OF L OSSES
Revised: A. A BLATION OF L OSSES

Original: To analyze the impacts of different losses, we visualize their t-SNE map in Fig.
Revised: To analyze the impacts of different losses, we visualize their t-SNE map in Fig.

Original: We can notice that just MSE loss cannot well align the feature of the two models.
Revised: We can notice that just MSE loss cannot well align the feature of the two models.

Original: The reconstruction loss is vital to keep discrimination and avoid overfitting.
Revised: The reconstruction loss is vital to keep discrimination and avoid overfitting.

Original: The adversarial loss seems not to show much improvement in the figure.
Revised: The adversarial loss seems not to show much improvement in the figure.

Original: According to our empirical study, adversarial loss only brings limited gains.
Revised: According to our empirical study, adversarial loss only brings limited gains.

Original: [Figure]
Revised: We have conducted the quantitative ablation study on a subset of COCO with 5k data randomly selected. We chose T5-3B as the text encoder aligned with the LDM Unet with our MTN. FT denotes the finetuning of Unet.

Original: 𝒎𝒔𝒆
Revised: We have reported the FID and CLIP scores in the following table for comparison of image quality and image-text alignment. The testing data are inferred by DDIM with 200 steps. The image size is 256 × 256. For a comprehensive analysis, the experiments are over

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Table]

Original: 𝓛 𝒓 𝒆 𝒄 𝒙" 𝒔𝒓𝒄 𝑴 𝑵
Revised: three classifier-free guidances with s =1.5, 5, and 7.5 according to Eq. The Tab 3 summarizes the results of our ablation study.

Original: M o d e l T r a n s l a , o n N e t w o r k
Revised: The first row represents the direct combination of T5 and LDM, whose results do not make sense due to the severe misalignment.

Original: ( M T
Revised: T5+LDM trained from scratch can be referred to in the second row. The finetuning of Unet takes 100 GPU Days, whereas MTN training needs only 5 GPU Days. Even with a ten times higher cost, its result is still inferior to those of MTNs’. By comparing the 3rd, 4th, 5th, and 6th rows, we can easily conclude the superiority of the full-version model.

Original: N ) 𝓛 𝒓 𝒆 𝒄 𝒙"# 𝒔𝒓𝒄 𝑴 𝑵
Revised: More analysis of hyper-parameters is included here.

Original: M o d e l T r a n s l a , o n N e t w o r k
Revised: The learning rate is one of the most important sensitive hyper-parameters.

Original: ( M T N ) 𝒙" 𝒔𝒓𝒄 𝓛 𝒂𝒍𝒊𝒈𝒏 …
Revised: In Tab. 4, we explored the different learning rates to try MTN to transfer

Revised: OpenClip-L to Clip-L for Stable Diffusion.

Revised: It is benchmarking on the average CLIP score withgenerative samples in the size of 512 × 512.

Original: A. C LOSED - LOOP C ONSISTENCY
Revised: A. C LOSED - LOOP C ONSISTENCY

Original: The closed-loop consistency is a helpful technique to accelerate MTN training.
Revised: The closed-loop consistency is a helpful technique to accelerate MTN training.

Original: We have briefly introduced it in Sec 4.1 in the main paper.
Revised: We have briefly introduced it in Sec 4.1 in the main paper.

Original: Here, we provide more details as illustrated in Fig. 
Revised: Here, we provide more details as illustrated in Fig. The Fig.

Original: The Fig. 13 and 14 visualize the curves of alignment and reconstruction losses during training.
Revised: 13 and A.5 visualize the curves of alignment and reconstruction losses during training.

Original: B M ONOLINGUAL T2I G ENERATION B. L ATENT D IFFUSION M ODEL
Revised: B M ONOLINGUAL T2I G ENERATION B. L ATENT D IFFUSION M ODEL

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 15 and 16, we demonstrate more example results with the challenging prompts from
Revised: 15 and 16, we demonstrate more example results with the challenging prompts from

Original: Drawbench (Saharia et al., 2022) and Winoground (Thrush et al., 2022).
Revised: Drawbench (Saharia et al., 2022) and Winoground (Thrush et al., 2022).

Original: The baseline methods are the original LDM and the finetuned LDM equipped with T5 as the text encoder.
Revised: The baseline methods are the original LDM and the finetuned LDM equipped with T5 as the text encoder.

Original: We can catch that the images of T5+MTN+FT are better aligned with the text and reasonable in the layout of different objects. 
Revised: We can catch that the images of T5+MTN+FT are better aligned with the text and reasonable in the layout of different objects. lr=1elr=1elr=1elr=1eCLIP ↑ 27.

Revised: 𝓛 𝒂𝒍𝒊𝒈𝒏 𝒙 𝒔𝒓𝒄 𝓛 𝒓 𝒆 𝒄 𝒙" 𝒔𝒓𝒄 𝑴 𝑵

Revised: M o d e l T r a n s l a , o n N e t w o r k

Revised: ( M T

Revised: N ) 𝓛 𝒓 𝒆 𝒄 𝒙"# 𝒔𝒓𝒄 𝑴 𝑵

Original:  B. S TABLE D IFFUSION M ODEL
Revised: M o d e l T r a n s l a , o n N e t w o r k ( M T N ) 𝒙 " 𝒔𝒓𝒄 𝓛 𝒂𝒍𝒊𝒈𝒏 … B. S TUDY OF D IFFERENT T EXT E NCODERS

Revised: our proposed framework is generic to a wide range of text encoders.

Revised: In this subsection, we have tried many different language models as the text encoder, including Bert-base (110M #params),

Revised: Roberta-Large (355M #params), Clip-text (123M #params), T5-large (770M #params) and T5-3B (2.8B

Revised: #params).

Revised: All these pre-trained text encoders are aligned with the Latent Diffusion Model (LDM) with our proposed model to replace the original LDM text encoder without any finetuning of the LDM UNet.

Revised: These plug-and-play models are verified on a subset of COCO with 5k samples randomly selected.

Revised: We have reported the FID and CLIP scores in the following table for comparison of image quality and image-text alignment.

Revised: All the model translation networks (MTN) are trained by 18M sentences sampled from the captions of Laion-400M.

Revised: The testing data are inferred by DDIM with 200 steps.

Revised: The image size is 256 × 256.

Revised: For a comprehensive analysis, we have conducted the experiments on three classifier-free guidance with s =1.5 and 5.

Revised: The results are summarized in Tab. 5.

Revised: From the table, we can notice that both FID and CLIP have increased with the rise of text model sizes which is well aligned with the conclusion of Imagen(Saharia et al., 2022).

Revised: The CLIP-text does not perform well here due to the larger domain gap with text-only text models such as Bert.

Revised: B. M ODEL T RANSFER ACROSS V ARIANT T OKEN L ENGTHS

Revised: MTN has a strong ability to overcome different lengths.

Revised: The token number is assigned as 77, which is fixed in both LDM and SDM.

Revised: Our proposed MTN can overcome the variant length text encoders without any finetuning of the Unet model.

Revised: We have completed the experimental study and reported its result in the Tab. 6 to verify this point.

Revised: In this experiment, we have applied Roberta-L and its tokenizers encoding maximum tokens in 77, 128, and 256, respectively.

Revised: The guidance is 5, and other settings follow the Tab. 3.

Original: To illustrate the generics of our proposed framework, we also apply the T5-L to replace the CLIP text encoder of Stable Diffusion (v1-4).
Revised: To illustrate the generics of our proposed framework, we also apply the T5-L to replace the CLIP text encoder of Stable Diffusion (v1-4).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 17, our model has revealed precise controllability and excellent visual quality compared with the standard Stable Diffusion model.
Revised: 17, our model has revealed precise controllability and excellent visual quality compared with the standard Stable Diffusion model.

Original: Improving
Revised: Improving

Original: Stable Diffusion with efficient finetuning is our following focus. 
Revised: Stable Diffusion with efficient finetuning is our following focus. [Figure] [Figure]

Revised: [Table]

Revised: We also have applied OpenCLIP-L-14 to replace CLIP-L-14 with the similar architecture on Stable

Original: [Figure]
Revised: Diffusion. The quantitative evaluation is reported in Tab. 7. This is tested on COCO-5k with the guidance as [Table]

Revised: The conditional generation itself is a very challenging task where how the text encoder aligns with image Unet still needs to be discovered.

Revised: We empirically find that replacing the text encoder can achieve the comparable results with the source model

Original: [Figure]
Revised: [Table]

Original: C M ULTILINGUAL T2I G ENERATION
Revised: but it takes a lot of effort to outperform it. The Unet may cause this as the bottleneck. In this section, we share a precise cost of computations with the different sizes of the training set in Tab. 8. This is benchmarked as the same as previous experiments on COCO-5K with T5-3B to replace the LDM text encoder with MTN.

Original: To verify the functionalities for multilingual T2I generation
Revised: From the table, we can see that the performance has increased with the rise of data as well as the computation cost. Moreover, we also include T5+LDM trained from scratch as a baseline. For a fair comparison, we have tried our best to train the T5+LDM with ∼ 140 GPU Days which is comparable with the sum of MTN116 and MTN116+FT. The results are shown in Tab. 9.

Original: [Figure]
Revised: [Table] [Table]

Revised: C M ULTILINGUAL T2I G ENERATION

Original:  we have shown more results in Fig.and Fig. 19.
Revised: To verify the functionalities for multilingual T2I generation, we have shown more results in Fig.and Fig. 19.

Original: There is one MTN for one language.
Revised: There is one MTN for one language.

Original: To build an automatic pipeline
Revised: To build an automatic pipeline, the model is expected to choose which MTN to use according to the detection of incoming languages.

Original: [Figure]
Revised: Our proposed framework is model agnostic which should be able to overcome variants domain gaps between models and languages. The cross-lingual alignment is applied as verification of proof of concept.

Original:  the model is expected to choose which MTN  to use according to the detection of incoming languages.
Revised: Compared with the standard translation + DM, our proposed MTN needs less training and fewer parameters to train. For quantitative evaluation, we have compared it with a popular multilingual translation model M2M100-418M (Fan et al., 2021)

Original: LDM Original
Revised: (418M #params) trained by 7.5B parallel sentences. In contrast, our MTN uses far fewer training data ( 2M sentences for each pair). The results are listed in Tab. 10 where we reported the Multilingual-CLIP score 3 calculated over the 100 randomly selected multilingual image-caption pairs from Crossmodal dataset (Thapliyal et al., 2022).

Revised: All the images are sampled by PLMS (Liu et al., 2022) with 50 steps in the resolution of× 512.

Revised: The cosine similarity of Multilingual-CLIP text and image embeddings is reported here.

Revised: From the Tab. 10, we can see that the MTN is only slightly weaker than M2M-418M despite the huge efficiency in training data.

Revised: Such a result reflects the great potential of MTN in multilingual T2I by aligning the cross-lingual model with the consideration of data efficiency.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: LDM Original

Original: T5+MTN T5+FT T5+MTN+FT
Revised: T5+MTN T5+FT T5+MTN+FT

Original: A green colored banana.
Revised: A green colored banana.

Original: A pink colored car.
Revised: A pink colored car.

Original: A stop sign on the right of a refrigerator.
Revised: A stop sign on the right of a refrigerator.

Original: An elephant under the sea.
Revised: An elephant under the sea.

Original: New York skyline with “hello world” written with fireworks on the sky.
Revised: New York skyline with “hello world” written with fireworks on the sky.

Original: A blanket is on a dog with spots.
Revised: A blanket is on a dog with spots.

Original: A child with a fake beard and an adult.
Revised: A child with a fake beard and an adult.

Original: A human being on a device that intercepts the sun’s rays.
Revised: A human being on a device that intercepts the sun’s rays.

Original: A parked car is partially inside a tree.
Revised: A parked car is partially inside a tree.

Original: A person is close to the water and in the sand.
Revised: A person is close to the water and in the sand.

Original: A person stands and a dog sits.
Revised: A person stands and a dog sits.

Original: Blue pants and green top.
Revised: Blue pants and green top.

Original: Wood ﬂoors with concretewalls.
Revised: Wood ﬂoors with concretewalls.

Original: LDM Original

Original: T

Original: [Figure]
Revised: [Figure]

Original: MTN

Original: [Figure] FT T [Figure]
Revised: Original

Original: MTN

Original: [Figure]
Revised: [Figure]

Original: FT
Revised: FT

Original: [Figure]

Original: Stable Diffusion (v1-4)
Revised: Stable Diffusion (v1-4)

Original: T5-L + MTN “An astronaut riding a horse.”
Revised: T5-L + MTN “An astronaut riding a horse.”

Original: “A pink car.”
Revised: “A pink car.”

Original: “A shark in the dessert.”
Revised: “A shark in the dessert.”

Original: [Figure] [Figure]
Revised: [Figure] [Figure]