Paper: Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR-2024)

Reviewer comment (Evaluation):
While the paper mentions techniques to reduce computational overhead by one-step back-propagation in the I2T2I cycle, it does not provide quantitative metrics to evaluate the efficiency of these techniques. For the pre-trained data used in the experiments, ITIT still uses 4M paired data and 398M unpaired data to compete with the method that uses 398M paired data. In that case, the overall computational overhead is larger than the traditional paired data training. Can you provide some analysis in comparing the training efficiency between the paired data training and the ITIT.

Author response:
**A1:** We include a detailed training speed analysis in Table 2 of the revised Appendix (new texts in purple). ITIT is ~2.32x slower than the no-cycle baseline during training (but the same speed as the baseline during inference). However, we note that the cost of doubling the training time is marginal compared to the cost of collecting large-scale paired datasets. We show in our paper that ITIT trained with only 4M paired data can achieve a similar performance as baseline trained with 400M paired data, reducing the cost of collecting paired data by 100 times.

Reviewer comment (Experiments):
The pseudo-synthesized data pairs could be noisy at the beginning of the training since the generator is not trained yet, I am wondering whether the pseudo-data pairs generated at the beginning will play negatively on the performance.

Author response:
**A2:** This is an interesting suggestion by the reviewer. We experimented with the two-stage training scheme with ITIT-B trained for 500K steps, starting the second stage training (cycle loss) at different training steps. As shown in Table 3 in the revised Appendix (new text in purple), starting the second stage at 25K steps achieves slightly better performance than starting from the beginning. Interestingly, introducing the cycle loss at the very beginning (0 steps) does not significantly impair performance. This resilience is likely due to the dominance of paired data in the early training stages. As the generation results improve, the impact of cycle consistency loss becomes more pronounced, achieving greater diversity and generalization ability through the utilization of unpaired data.

Reviewer comment (Theory):
Question for the gradient estimation of the cycle training: Referring to the figure 3, T2I is performed twice with a stop gradient operation in the middle for the T2I2T, why not directly perform the T2I once. Similar questions for the twice I2T in the process of I2T2I.

Author response:
**A3:** In ITIT, the synthesized image and text need to be of reasonable quality to help the cycle training. However, existing T2I and I2T methods cannot generate high-quality images or texts in one step. Therefore, we need to first generate the images and texts in multiple steps (24 steps for T2I and auto-regressively for I2T) to ensure it is of reasonable quality, and then back-propagate the cycle-consistency loss to one of these generation steps.

Reviewer comment (Reproducibility):
Why ViT is chosen as the joint image-text encoder, instead of using some multimodal architectures. How does ViT extract the text feature?

Author response:
**A4:** Before the image-text encoder, we use a pre-trained T5 encoder to extract text embeddings from raw text tokens. The text embeddings are then concatenated with the image token embeddings as the input to our image-text encoder. Conceptually, our image-text encoder could use any multimodal network architecture. For simplicity, we use a simple architecture consisting of a stack of Transformer blocks, where each block consists of a multi-head self-attention block and an MLP block. We call it a “ViT” because it takes in both visual and text inputs, but it uses a general architecture that is not specific to visual inputs. We will clarify this term in the revised manuscript.

Reviewer comment (Evaluation):
The author only mentions a reduction in memory usage, but the time consumption should be quite large, which is a disadvantage. Any justification for this?

Author response:
**Q1: Training time**

**A1:** We include a detailed training speed analysis in Table 2 of the revised Appendix (new texts in purple). ITIT is ~2.32x slower than the no-cycle baseline during training (but the same speed as the baseline during inference). However, the cost of doubling the training time is acceptable compared to the huge cost of collecting large-scale high-quality paired datasets. We show in our paper that ITIT trained with only 4M paired data can achieve a similar performance as baseline trained with 400M paired data, reducing the cost of collecting paired data by 100 times.

Reviewer comment (Theory):
The I2T2I method seems a bit unreasonable because the latter half of T2I can already generate many kinds of images, so why must reconstructing the image necessarily be correct?

Author response:
**Q2: Why must reconstructing the image**

**A2:** Our I2T2I reconstruction loss is a cross-entropy loss between reconstructed image tokens and ground-truth image tokens. Such a cross-entropy-based reconstruction loss is widely used in many prior image generation works [1, 2, 3]. Instead of reconstructing a single image as the reviewer’s question implied, the cross-entropy loss asks the model to maximize the log-likelihood of the input image. For example, multiple dog images could be mapped to the same text “dog”. The T2I model is then asked to reconstruct the image tokens from the text prompt “dog” so that it maximizes the log-likelihood of all the dog images.

A similar cross-entropy-based reconstruction loss is also widely employed in many other tasks. One of them is training large language models. During training, the network is trained to predict the next token in the ground-truth sentence given the current tokens, which could have multiple possibilities. The loss is a cross-entropy loss between the reconstructed text token and the ground-truth text token, which helps the LLM to maximize the log-likelihood of the next token.

Reviewer comment (Experiments):
For visual-language cycle consistency, there are two papers already: 'Equivariant Similarity for Vision-Language Foundation Models' (ICCV 2023), and 'CyCLIP: Cyclic Contrastive Language-Image Pretraining' (NeurIPS 2022). In this submission, there is an image-text encoder, for which it’s very straightforward to use the above two methods as a baseline (in the cycle consistency of embedding). So the reviewer is quite curious about how the cycle of embedding compares to the cycle of generating data.

Author response:
**Q3: Prior works in visual-language cycle consistency**

**A3:** We appreciate the reviewer's effort in highlighting relevant literature. Unlike these papers, which concentrate on visual representation learning, ITIT is dedicated to image-to-text (I2T) and text-to-image (T2I) generation tasks. Furthermore, while these papers utilize text supervision to enhance visual representations through image-text similarity regularizations, they rely on **paired** image-text data for this purpose, as evident in Equation (6) of the ICCV 2023 paper [6]. In contrast, ITIT employs cycle consistency to leverage **unpaired** data for vision-language generative pre-training.

We have also incorporated a discussion of these papers in the related work section of our revised manuscript (new texts in purple). It's important to note that our current image-text encoder is not pre-trained using any image-text representation learning methods, ensuring a fair comparison with prior works [1, 2]. An intriguing future research would be to explore the use of an encoder pre-trained with methods such as CLIP [4] as well as the papers suggested by the reviewer ([5, 6]) as the initialization for our image-text encoder.

**References**

[1] Huiwen Chang, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023.

[2] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and
instruction tuning. arXiv preprint arXiv:2309.02591, 2023.

[3] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873–12883, 2021.

[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.|

[5] Goel, Shashank, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

[6] Wang, Tan, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. "Equivariant Similarity for Vision-Language Foundation Models." arXiv preprint arXiv:2303.14465 (2023).

Reviewer comment (Evaluation):
In Table 2, the gap between full cycle and half cycle of ITIT training seems very marginal, which raises doubts about the effectiveness of proposed consistency concept. Namely, it seems that the pseudo-targets (ITIT-half cycle) would be sufficient to leverage unpaired image and texts. In this case, I believe that generating pseudo-targets would be more powerful: 1) it is computationally efficient and 2) it could easily integrate with the recent pre-trained models such as BLIP v2, stable-diffusion, and so on. (since the generation process of pseudo-targets are model-agnostic)

Author response:
**A1:** We are grateful for the reviewer's recognition of the effectiveness of our half-cycle training in achieving notable performance. Indeed, the core objective of the ITIT framework is to harness unpaired image and text data for vision-language generative training. Through our experiments, we demonstrate that both half-cycle and full-cycle iterations of ITIT are capable of leveraging unpaired data to significantly enhance image-to-text (I2T) and text-to-image (T2I) generation.

As detailed in Table 2 of the main paper and Table 2 of the revised appendix (new texts in purple), it is evident that full-cycle training offers additional improvements over half-cycle training. Notably, this enhancement comes with only a marginal increase in training speed overhead (around 10%) and no additional inference overhead. The performance of full-cycle training can be attributed to its more rigorous enforcement of cycle consistency, as illustrated in Figure 6 of the main paper.

Conversely, we acknowledge that implementing half-cycle training with other pre-training methods may present a more straightforward approach. This consideration reflects a trade-off inherent in the two versions of ITIT: while full-cycle training offers higher performance due to its comprehensive cycle consistency, half-cycle training provides greater flexibility and ease of adaptation to various pre-training methodologies. This balance between flexibility and performance efficiency is a key aspect of our ITIT framework's design and application.

Reviewer comment (Evaluation):
Given that ITIT employs synthetically generated captions or images during the training process, there is a reasonable expectation that numerous failure cases may occur. This concern is particularly pronounced with datasets like CC3M, which are known to consist of noisy image-text pairs, potentially leading to a higher frequency of failures. It would be grateful if authors could share the fair cases.

Author response:
**A2:** We thank the reviewer for the suggestion. We added some failure cases in Fig. 5 (page 9)  of the revised Appendix. As shown in the figure, ITIT encounters failure cases that are common across most image-to-text generation models. Some of the issues, such as watermarks, are closely tied to the quality of the data utilized in training. Given this context, we believe that ITIT has the potential to mitigate these prevalent shortcomings by lessening the dependency on paired data in vision-language generative training. A promising direction for future research would involve training ITIT with a combination of a small but exceptionally high-quality, paired image-text dataset and extensive, high-quality unpaired image and text datasets.

Reviewer comment (Evaluation):
This paper claims that the paired data gap is almost closed in Figure 4 (from 398M to 4M). However, there is still a significant gap in FID and CIDEr according to Figure 1 compared with other methods e.g. CM3Leon, Muse, GIT. Is there any other reason besides different numbers of parameters? Or maybe Shutterstock data is so similar that scaling data does not make a difference, but on more diverse datasets the Figure 4 claim will not hold?

Author response:
We acknowledge the reviewer's point regarding the comparison between ITIT and existing text-to-image (T2I) and image-to-text (I2T) methods, such as CM3Leon, Muse-512, and GIT. It is important to note that these methods are trained with significantly more parameters (3B for Muse and 7B for CM3Leon) and paired data (860M for Muse and 360M for CM3Leon) than our ITIT model. GIT is also trained on a substantially larger dataset (800M), employs a pre-trained CLIP model for initializing its image encoder (as opposed to training from scratch), and incorporates the COCO training dataset in its training regime, the test set of which is used for evaluation. Therefore, drawing a direct performance comparison between ITIT and these methods would be inequitable.

In Table 1 of our paper, we present the results of training GIT and Muse under conditions more comparable to those of ITIT. Under similar constraints of using the same paired dataset (CC3M) and a comparable number of network parameters, ITIT demonstrates superior performance in terms of both FID and CIDEr scores. This improvement is attributed to the effective leveraging of unpaired data from WebLI, facilitated by the cycle consistency loss.

It's crucial to point out that the experiments in Table 1 and Figure 4 utilize different datasets. Figure 4 employs Shutterstock for both paired and unpaired data, whereas the results in Table 1 are derived from using CC3M as the paired data source and WebLI as the unpaired one. Given that CC3M exhibits the highest image-text pairing quality among the datasets we considered, ITIT achieves better performance on the CC3M+WebLI combination compared to Shutterstock.

The primary focus of Figure 4 is to investigate how ITIT enhances performance over the baseline using the same amount of paired data augmented with additional unpaired data. To effectively study this, the added unpaired data should not present a domain gap with the paired data, rendering the CC3M+WebLI setting impractical for this specific analysis. Therefore, we opted to conduct this part of the experiment using Shutterstock. This choice also demonstrates that the cycle consistency mechanism is beneficial across different datasets, further highlighting its versatility and effectiveness.

Reviewer comment (Experiments):
How does ITIT generalize to new/different domains without paired data in that domain (assuming unpaired domain specific text and image available)? One example readers might be interested in could be medical imaging where paired data is much more difficult to obtain.

Author response:
We believe that our ITIT model has the potential to significantly reduce the reliance on extensive paired data in specialized fields like medical imaging. This reduction in dependence on paired data could open up new possibilities for domain adaptation, where paired data from one domain might facilitate effective model training using unpaired data from other domains. Exploring this possibility of cross-domain adaptation using ITIT represents an exciting and promising avenue for future research.

Reviewer comment (Experiments):
Unclear compatibility with other types of generative models such as diffusion/flow/consistency models as the I2T stage. How much does the ITIT framework depends on the unified architecture? Does ITIT work with any kind of T2I I2T model?

Author response:
ITIT represents a general vision-language generative training scheme, adaptable to various text-to-image (T2I) and image-to-text (I2T) frameworks. The core principle of employing cycle consistency loss to effectively utilize unpaired image and text data is universally applicable. Nonetheless, an essential aspect of the ITIT framework is its ability to perform T2I generation in an online manner, necessitating a focus on inference speed when selecting a suitable T2I framework.

In our study, we choose a parallel decoding T2I framework similar to Muse [1], known for its efficiency in T2I generation. Our proposed unified image-text encoder further emphasizes simplicity and reduced training costs. It's important to clarify that these attributes are not obligatory for the implementation of ITIT. The framework is flexible enough to accommodate other fast inference models, such as latent diffusion [2] or latent consistency models [3]. However, we note that the primary aim of our paper is to demonstrate the potential and advantages of incorporating unpaired data into vision-language training. The integration of other text-to-image (T2I) and image-to-text (I2T) frameworks, while equally important, is designated for future research endeavors.

Reviewer comment (Experiments):
ITIT essentially denoises training data by learning a better alignment as shown in Figure 6. Does it worth to study other methods that also cleans the data, e.g. simply CLIP filtering with high confidence as pseudo labels, i.e. semi-supervised discriminative learning? This might keep the “blue” information in Figure 6 as well?

Author response:
The primary objective of our ITIT framework is to effectively utilize unpaired image and text data within the context of vision-language generative training. In contrast, existing data cleaning methods, like CLIP filtering, are designed to refine noisy paired datasets, enhancing the quality of image-text pairings. These methods, however, are not suited for handling unpaired data. Consequently, ITIT should be seen as complementary to data filtering approaches. While the latter focuses on elevating the quality of paired data, ITIT is designed to harness the potential of unpaired data. This distinction underscores ITIT's unique contribution to the field, offering a novel approach that broadens the scope of vision-language training beyond the confines of traditional paired data methodologies.

Reviewer comment (Reproducibility):
Runtime due to the pseudo pair generation stage. The appendix mentioned that the ITIT requires 2x training time compared with T2I and I2T non-cycle training, but how is it the case considering the 24 steps of decoding happening? How much of the time was spent on the online pseudo pair generation stage? Does it require careful batching or implementation for good TPU utilization?

Author response:
In the revised appendix of our paper (new texts in purple), we have included a comprehensive training cost analysis, presented in Table 2. In the scenario of T2I2T half-cycle training (row 2), we observe that the speed is 1.4 times slower than that of the no-cycle approach (row 1). Specifically, for row 1, a batch size of 1024 is allocated for T2I training, while the remaining 1024 is used for I2T/I2I training. In contrast, row 2 involves 512 data for T2I, 512 for I2T/I2I, and an additional 1024 dedicated to T2I2T training.

The primary reason for this increased training time is the 24-step T2I online generation process. However, it's important to note that a significant portion of the training time is consumed by the gradient backpropagation step, which is the same for no-cycle and half-cycle cycle training and slightly higher for full-cylce training. Consequently, the additional time required for the online generation phase is considered acceptable within the overall training process.

Reviewer comment (Experiments):
Although scaling data is left to future work, the value of this paper depends a lot on the promise of further scaling unpaired image and text data beyond the combination of all current paired datasets. Are we still able to collect 10x high quality images/texts but not paired? Do we still need more data in this domain? Unfortunately this has not been shown to a good extent.

Author response:
This is a great question which we believe should be investigated further. As shown in many recent works, the model and data size of vision-language generative models are far from saturated, and the quality of image-text pairing is also crucial in generation performance [4, 5, 6, 7]. However, paired data is much harder to collect than unpaired data. Pali [5], for example, shows that they are able to collect 10 billion images and 12 billion texts from the internet, but after image-text alignment filtering, only 10% of the image-text pairs are kept. Therefore, ITIT has great potential in utilizing unpaired image and text data to improve the performance of vision-language generative models. Moreover, we believe that the idea of utilizing unpaired data in multimodal generative training could be broadly applied to other modalities where collecting paired data is even harder, such as medical imaging or disease diagnosis.