Paper: CoBIT: A Contrastive Bi-directional Image-Text Generation Model (ICLR-2024)

Reviewer comment (Theory):
Assumed Synergy: The core premise of CoBIT hinges on the idea of a symbiotic relationship between image-to-text and text-to-image generation tasks. However, the paper doesn't thoroughly investigate or justify the assumed synergy. It's crucial to establish a theoretical foundation for this assumption, or the unified framework might not hold in different contexts or datasets.

Author response:
**Answer**: We really appreciate your excellent suggestion. Based on the ablation in Sec. 4.4, we would like to provide further analysis of the mutual effects of three different losses:


* **Finding**: Cross-modal generation objectives can improve image understanding a bit on top of contrastive loss. **Analysis**: Either T2I objective or I2T objective focuses on cross-modal fine-grained semantics, which can explicitly facilitate the global alignment between two modalities.

* **Finding**: Two generation losses, i.e., I2T loss and T2I loss, contradict each other a little bit. **Analysis**: The good signal is that those two don’t contradict that much and still can keep a comparable performance when jointly training. It’s promising because joint training essentially saves half of the parameters compared with using an ensemble of two separate models. On the other hand, those two losses don’t show mutual benefit in performance as we expected. We think the reason is that the granularity of image generation and text generation are different. A single word like cat in the text may correspond to a region in the image that contains hundreds of pixels and tens of image tokens. 

* **Finding**: Contrastive loss improves vision-language understanding while it doesn’t influence image generation. **Analysis**: A good global alignment enabled by contrast provides the base for better image-to-text generation. However, image generation requires precise local details beyond just correct semantics, which makes it hard to benefit from global alignment. 

As you can see, the mutual effects are kind of mixed, but the parameter efficiency of joint training always holds, i.e., the number of parameters in the ensemble of three specialized models is 2-3 times that of a jointly trained model. 

As for a more theoretical justification, we think existing works [1] about gradient harmony in multi-lingual model training might provide a new angle to our scenario. We two share the similarity in terms of multiple objectives, large data size, large model size, and intuitively mutual benefits of different objectives (vision is believed to be another language). Studying how the gradients of those three objectives interact should be more fundamental and theoretical, but also sophisticated. We would like to leave it as the next step.

[1] Wang, Zirui, et al. "Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models." arXiv preprint arXiv:2010.05874 (2020).

Reviewer comment (Theory):
Objective Conflicts: While unifying different objectives under a single framework is innovative, it poses a risk of conflicting objectives that might detract from optimizing each task individually. The paper acknowledges this to some extent but doesn't provide a robust solution to mitigate potential conflicts.

Author response:
**Answer**: Thanks for your valuable suggestion. We do consider the design of the proposed Unicoder-Decoder an initial exploration of the conflict mitigation strategies. From the ablation result, when jointly training with different objectives, it can outperform conventional encoder-decoder models. Intuitively, it shares the knowledge between encoding a modality for cross-modality understanding and generating a modality from the other modality, which explicitly bridges both understanding loss and generation loss by sharing the parameter space. We think there is still plenty of space to explore and believe our findings can shed light on future works.

Reviewer comment (Evaluation):
Evaluation Scope: The evaluation primarily focuses on showcasing the model's strengths, with a less thorough investigation into the model’s weaknesses or failure modes. A more balanced evaluation, including a deeper exploration of where the model falls short, would provide a more comprehensive understanding of the model's capabilities and limitations.

Author response:
**Answer**: Thank you for pointing out. In Sec. 4.5 and Sec. 6.6, we conducted some failure analysis about image generation. Several observations were presented and we quote them here. We also included the visualization examples for each type of failure in the manuscript.


* CoBIT sometimes messes up the size attributes of two objects.

* CoBIT sometimes couldn’t render the details of words in text very well.

* CoBIT occasionally misunderstands the text.


Here we further add a brief failure analysis of image-to-text generation in zeroshot scenario:

* CoBIT tends to generate short text, for example, one sentence or multiple phrases. This might be due to the training data distribution.

* CoBIT sometimes cannot follow instructions but instead tends to complete the input text. Additionally, instruction-tuning should help. 

Based on the above points, we will provide a more detailed anaylsis to help future improvement in the camera-ready version.

Reviewer comment (Theory):
Theoretical Justification: A deeper theoretical analysis of the assumed synergy between image-to-text and text-to-image generation tasks could strengthen the premise of the unified framework.

Author response:
**Answer**: We really appreciate your excellent suggestion. Based on the ablation in Sec. 4.4, we would like to provide further analysis of the mutual effects of three different losses:


* **Finding**: Cross-modal generation objectives can improve image understanding a bit on top of contrastive loss. **Analysis**: Either T2I objective or I2T objective focuses on cross-modal fine-grained semantics, which can explicitly facilitate the global alignment between two modalities.

* **Finding**: Two generation losses, i.e., I2T loss and T2I loss, contradict each other a little bit. **Analysis**: The good signal is that those two don’t contradict that much and still can keep a comparable performance when jointly training. It’s promising because joint training essentially saves half of the parameters compared with using an ensemble of two separate models. On the other hand, those two losses don’t show mutual benefit in performance as we expected. We think the reason is that the granularity of image generation and text generation are different. A single word like cat in the text may correspond to a region in the image that contains hundreds of pixels and tens of image tokens. 

* **Finding**: Contrastive loss improves vision-language understanding while it doesn’t influence image generation. **Analysis**: A good global alignment enabled by contrast provides the base for better image-to-text generation. However, image generation requires precise local details beyond just correct semantics, which makes it hard to benefit from global alignment. 

As you can see, the mutual effects are kind of mixed, but the parameter efficiency of joint training always holds, i.e., the number of parameters in the ensemble of three specialized models is 2-3 times that of a jointly trained model. 

As for a more theoretical justification, we think existing works [1] about gradient harmony in multi-lingual model training might provide a new angle to our scenario. We two share the similarity in terms of multiple objectives, large data size, large model size, and intuitively mutual benefits of different objectives (vision is believed to be another language). Studying how the gradients of those three objectives interact should be more fundamental and theoretical, but also sophisticated. We would like to leave it as the next step.

[1] Wang, Zirui, et al. "Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models." arXiv preprint arXiv:2010.05874 (2020).

Reviewer comment (Experiments):
Conflict Mitigation Strategies: Developing and incorporating strategies to mitigate the potential conflicts between different objectives could help in achieving a more balanced optimization across all tasks.

Author response:
**Answer**: Thanks for your valuable suggestion. We do consider the design of the proposed Unicoder-Decoder an initial exploration of the conflict mitigation strategies. From the ablation result, when jointly training with different objectives, it can outperform conventional encoder-decoder models. Intuitively, it shares the knowledge between encoding a modality for cross-modality understanding and generating a modality from the other modality, which explicitly bridges both understanding loss and generation loss by sharing the parameter space. We think there is still plenty of space to explore and believe our findings can shed light on future works.

Reviewer comment (Evaluation):
Failure Analysis: Conducting a thorough failure analysis to identify the model's weaknesses and understanding its behavior under different conditions or limitations could provide a clearer path for future improvements.

Author response:
**Answer**: Thank you for pointing out. In Sec. 4.5 and Sec. 6.6, we conducted some failure analysis about image generation. Several observations were presented and we quote them here. We also included the visualization examples for each type of failure in the manuscript.


* CoBIT sometimes messes up the size attributes of two objects.

* CoBIT sometimes couldn’t render the details of words in text very well.

* CoBIT occasionally misunderstands the text.


Here we further add a brief failure analysis of image-to-text generation in zeroshot scenario:

* CoBIT tends to generate short text, for example, one sentence or multiple phrases. This might be due to the training data distribution.

* CoBIT sometimes cannot follow instructions but instead tends to complete the input text. Additionally, instruction-tuning should help. 

Based on the above points, we will provide a more detailed anaylsis to help future improvement in the camera-ready version.

Reviewer comment (Novelty):
The whole pipeline is not novel and is more like an assembling of existing method/components. As mentioned by the authors, the unicoder is already used in other works (Sec. 3.2). Three objectives for cross-modal learning are commonly used. Even though it is first joint learned, I cannot see the advantage of combining them. Although the authors claim that they should benefit from each other, it is intuitive that they have different focuses. For example, contrastive loss will benefit retrieval tasks. It could be better if the authors could have more insights on each objective’s advantages from their results.

Author response:
> **W1: Insights on each objective’s advantages.**

**Answer**: Thank you for your valuable suggestion. Here we provide a more detailed explanation of each objective’s effects from their results. 

First of all, the parameter efficiency of joint training always holds, i.e., the number of parameters in the ensemble of three specialized models is 2-3 times that of a jointly trained model. Second, we don’t intend to make a point that those three objectives must benefit each other, but rather we want to serve as the first exploration about the relationship of those three objectives. In the following, we analyze the three findings we stated in Sec. 4.4 of original submission. 

* **Finding1**: Cross-modal generation objectives can improve image understanding a bit on top of contrastive loss. **Analysis**: Either T2I objective or I2T objective focuses on cross-modal fine-grained semantics, which can implicitly facilitate the global alignment between two modalities.

* **Finding2**: Two generation losses, i.e., I2T loss and T2I loss, contradict each other a little bit. **Analysis**: The good sign is that those two don’t contradict that much and still can keep a comparable performance when jointly training. It’s promising because joint training essentially saves half of the parameters compared with using an ensemble of two separate models. On the other hand, those two losses don’t show mutual benefit in performance as we expected. We think the reason is that the granularity of image generation and text generation are different. A single word like cat in the text may correspond to a region in the image that contains hundreds of pixels and tens of image tokens. 

* **Finding3**: Contrastive loss improves vision-language understanding while it doesn’t influence image generation. **Analysis**: A good global alignment enabled by contrast provides the base for better image-to-text generation. However, image generation requires precise local details beyond just correct semantics, which makes it hard to benefit from global alignment. 

As you can see, the mutual effects are kind of mixed in terms of performance, but overall those three losses are compatible in the unified framework. We believe that there is plenty of space for better unifying these objectives. And our proposed Unicoder-Decoder provides an option for model design improvement.

Reviewer comment (Experiments):
The advantages of the model over other models are not clear. There is no performance that shows fair comparison with other models. It could be the dataset used for pre-training, batch-size, pretraining epochs, etc.

Author response:
> **W2: Comparison w/ other models.**

**Answer**: Thank you for your feedback. Comparing with other methods in an absolutely fair environment is not realistic because many works use their own private datasets, such as CLIP, Florence, DALLE, etc, and different models have different model sizes, batch sizes, and training steps. 
Since jointly learning cross-modal contrastive optimization, text-to-image generation, and image-to-text generation is a new setting, ours serves as an initial exploration and mainly studies how those three objectives impact each other, and which model structure is more effective and efficient (Unicoder-Decoder vs. Encoder-Decoder). On those two core problems, all our experiments and ablations are **conducted fairly with the same amount of data**. So the findings of our paper should be valid and useful to the community.

Reviewer comment (Experiments):
Can you list the exact parameter sizes of each model in Table 1, 2, 3? That should be more fair comparison.

Author response:
> **Questions: Can you list the exact parameter sizes of each model in Table1,2,3? That should be more fair comparison. • Why are the tasks shown in ablation studies (Table 4,5,6) not consistent? What is the reason for lacking some tasks in each table? • Please check the weakness for more questions.**

**Answer**:  In Tab1, we already list the number of parameters of the models. As for models in Tab2, here are the exact parameter sizes of each model:

| Model | CoBIT-Base | CoBIT-Large | CLIP | ALIGN | FILIP | Florence | CoCa-Large | ZeroCap | SimVLM | VLKD | Parti-350M | Parti-750M | LDM (SD) | Coca-2B | Make-A-Scene | Versatile Diffusion | CoDi | DALL-E 2 | CM3Leon-7B | Parti-20B |
|-------|:----------:|:-----------:|:----:|:-----:|:-----:|:--------:|:----------:|:-------:|:------:|:----:|:----------:|:----------:|:--------:|:-------:|:------------:|---------------------|------|----------|------------|-----------|
| Size  |    554M    |    1082M    | 428M |  820M |  428M |   893M   |    787M    |   431M  |  632M  | 713M |    350M    |    750M    |   1.4B   |   2.1B  |      4B      | 3.4B                | 4B   | 3.5B     | 7B         | 20B       |

Here are the exact parameter sizes of each model in Tab3:

| Model | CoBIT-Base | CoBIT-Large | CLIP | ALIGN | UNITER | VinVL | CLIP-ViL | ALBEF | BLIP | SimVLM |  OFA | X-LXMERT | Unified-IOXL | CoDi | CoCa-2.1B | BEIT3 | PALI | Parti |
|-------|:----------:|:-----------:|:----:|:-----:|:------:|:-----:|:--------:|:-----:|:----:|:------:|:----:|:--------:|:------------:|:----:|:---------:|-------|------|-------|
| Size  |    554M    |    1082M    | 428M |  820M |  303M  |  340M |   136M   |  210M |  417 |  632M  | 930M |   227M   |     2.9B     |  4B  |    2.1B   | 1.9B  | 17B  | 20B   |

In ablations, in default, we use zeroshot ImageNet classification, VQA and zeroshot text-to-image generation as three representative tasks. But when comparing Unicoder-Decoder vs. Encoder-Decoder, finetuning on VQA task updates more parameters in Unicoder-Decoder than Encoder-Decoder, which is not fully fair. To further clarify, we added the zeroshot image captioning task as a fair evaluation for those two designs and have to delete zeroshot ImageNet classification to save space. The performance of zeroshot ImageNet classification is improved by 0.7% when comparing Unicoder-Decoder to Encoder-Decoder. We will add all the details to the camera-ready version.

Reviewer comment (Experiments):
Why are the tasks shown in ablation studies (Table 4, 5, 6) not consistent? What is the reason for lacking some tasks in each table?

Author response:
> **Questions: Can you list the exact parameter sizes of each model in Table1,2,3? That should be more fair comparison. • Why are the tasks shown in ablation studies (Table 4,5,6) not consistent? What is the reason for lacking some tasks in each table? • Please check the weakness for more questions.**

**Answer**:  In Tab1, we already list the number of parameters of the models. As for models in Tab2, here are the exact parameter sizes of each model:

| Model | CoBIT-Base | CoBIT-Large | CLIP | ALIGN | FILIP | Florence | CoCa-Large | ZeroCap | SimVLM | VLKD | Parti-350M | Parti-750M | LDM (SD) | Coca-2B | Make-A-Scene | Versatile Diffusion | CoDi | DALL-E 2 | CM3Leon-7B | Parti-20B |
|-------|:----------:|:-----------:|:----:|:-----:|:-----:|:--------:|:----------:|:-------:|:------:|:----:|:----------:|:----------:|:--------:|:-------:|:------------:|---------------------|------|----------|------------|-----------|
| Size  |    554M    |    1082M    | 428M |  820M |  428M |   893M   |    787M    |   431M  |  632M  | 713M |    350M    |    750M    |   1.4B   |   2.1B  |      4B      | 3.4B                | 4B   | 3.5B     | 7B         | 20B       |

Here are the exact parameter sizes of each model in Tab3:

| Model | CoBIT-Base | CoBIT-Large | CLIP | ALIGN | UNITER | VinVL | CLIP-ViL | ALBEF | BLIP | SimVLM |  OFA | X-LXMERT | Unified-IOXL | CoDi | CoCa-2.1B | BEIT3 | PALI | Parti |
|-------|:----------:|:-----------:|:----:|:-----:|:------:|:-----:|:--------:|:-----:|:----:|:------:|:----:|:--------:|:------------:|:----:|:---------:|-------|------|-------|
| Size  |    554M    |    1082M    | 428M |  820M |  303M  |  340M |   136M   |  210M |  417 |  632M  | 930M |   227M   |     2.9B     |  4B  |    2.1B   | 1.9B  | 17B  | 20B   |

In ablations, in default, we use zeroshot ImageNet classification, VQA and zeroshot text-to-image generation as three representative tasks. But when comparing Unicoder-Decoder vs. Encoder-Decoder, finetuning on VQA task updates more parameters in Unicoder-Decoder than Encoder-Decoder, which is not fully fair. To further clarify, we added the zeroshot image captioning task as a fair evaluation for those two designs and have to delete zeroshot ImageNet classification to save space. The performance of zeroshot ImageNet classification is improved by 0.7% when comparing Unicoder-Decoder to Encoder-Decoder. We will add all the details to the camera-ready version.

Reviewer comment (Experiments):
Computational Efficiency: The use of Unicoder variants in cross-modal generation scenarios, as mentioned in the appendix of the paper, appears to increase the computational load and the number of parameters used in downstream fine-tuning tasks. However, this aspect is not well elaborated in the main text of the paper. A more thorough discussion on the computational efficiency and trade-offs of using Unicoder compared to Encoder would provide clearer insights into the practicality of the model.

Author response:
Thank you for your suggestion. Yes, in training, Unicoder-Decoder and Encoder-Decoder models consume the same number of parameters; in the finetuning of text-to-image and image-to-text tasks, Unicoder-Decoder inherits more parameters than Encoder-Deocder. That is exactly the reason why we want to evaluate zeroshot captioning and zeroshot image generation in the ablation experiments (Tab. 5), which can eliminate the factor brought by updating additional parameters. In those two tasks, Unicoder-Deocoder still outperforms Encoder-Decoder, which validates its effectiveness. We will add this illustration in the camera-ready version of our paper. 

Here we provided a comprehensive table about the number of parameters, training time, and performance of the main model variants in our paper. We will further verify the inference speed and memory cost precisely and consider adding them in the final version. 

|                                                  | # Parameters | Training Time (200k steps, shrunken bz) | ZS ImageNet Classification. | VQA      | ZS Caption | ZS Image Generation. (fid, the lower the better) |
|--------------------------------------------------|--------------|-----------------------------------------|-----------------------------|----------|------------|--------------------------------------------------|
| Contrastive Loss Only                            | 229M         | 70 TPUv4 Days                           | 70.8                        |          |            |                                                  |
| Image-to-Text Loss Only                          | 526M         | 91 TPUv4 Days                           |                             | 68       | -          |                                                  |
| Text-to-Image Loss Only                          | 418M         | 96 TPUv4 Days                           |                             |          |            | 12.6                                             |
| Ensemble of Three Separate Models                | 1173M        | 257 TPUv4 Days                          | 70.8                        | 68       | -          | 12.6                                             |
| **CoBIT (Joint training w/ Three losses)**       | **554M**     | **139 TPUv4 Days**                      | **71.1**                    | **66.9** | **37.9**   | **13.3**                                         |
| Encoder-Decoder (Joint training w/ Three losses) | 554M         | 135 TPUv4 Days                          | 70.4                        | 65.9     | 32.9       | 13.8                                             |

Reviewer comment (Experiments):
Impact of Pre-training Data: The paper shows good ImageNet linear probe performance for CoBIT in Table 3. It would be interesting to see a more detailed analysis of how the pre-training data impacts the model's performance. This could include experiments with different sizes or types of pre-training datasets.

Author response:
Thank you for this valuable suggestion. Here we listed the ablation on the pre-training datasets. To make a fair comparison, the batch size is kept the same, the training is conducted for 200k steps on base model. As we can see, JFT is beneficial to classification tasks, focusing on basic and precise semantics of image; WebLI has higher-quality image data and is specifically beneficial to text-to-image generation; ALIGN is relatively noisy but covers broad semantics. That’s the reason why we mixed them for a more balanced training.

|                    | ZS. IN. | LP. IN. |  VQA | ZS. T2I (fid, the lower the better) |
|--------------------|:-------:|:-------:|:----:|:-----------------------------------:|
| JFT                |   71.6  |   81.4  | 64.8 |                 14.6                |
| ALIGN              |   70.9  |    81   | 67.2 |                 13.8                |
| WebLI              |   70.0  |   80.2  | 66.2 |                 13.4                |

Thank you so much for pointing it out. There might be multiple reasons for this. One potential hypothesis is that the T2I task requires diverse semantics and might be more data-hungry, thus having more non-repeated data samples covering a wider range of semantics might offset its data quality issue (in ALIGN/JFT) and provide more generalizability when mixed with the high-quality data (WebLI). We will add more analysis and discussion about it in the camera-ready version if accepted.

Reviewer comment (Experiments):
Ablation Study: The ablation study in the paper could be strengthened.

Author response:
We will add the ablation about the loss weights in the appendix and the new ablation about different types of pre-training datasets in the main paper in the camera-ready version.

Thank you for your feedback. So far, we presented the ablations on **Mutual Effect of Three Objectives**, **Unicoder-Decoder vs. Encoder-Decoder**, **Initialization**, **Co-efficients of Losses**, **Pre-train Data**. If there is any specific ablation study you think can strengthen the paper, we are happy to add it in the later version.

Reviewer comment (Experiments):
From Table 1, it can be observed that the proposed base and large sizes of the model have more parameters compared to other multimodal models. This increase in model complexity might have implications for computational resources and efficiency. The authors should provide more explanation or justification for this design choice, and possibly discuss the trade-off between model complexity and performance.

Author response:
In previous multimodal works, different models have different numbers of parameters for their **base** or **large** models. For example, Coca’s base model has 383M while GIT’s base model has 129M. The reason why our base/large models have a bit more parameters than the previous works’ base or large design is that we have three objectives covering a large number of downstream tasks. Nevertheless, in Tab.2 and Tab.3, we can see that, CoBIT, as a multi-task model, can outperform most of the specialized models within 1B parameters and can even outperform larger models.

UniLMs only have one modality as input and output, and the tasks are all NLP-related. However, in CoBIT, two modalities are considered in both input and output, enabling CoBIT to tackle more tasks across modalities, which naturally requires more capabilities and parameters, especially T2I task requires more parameters to generate detailed images (we can find that the SOTA T2I models in Tab2,3 have more parameters on average).

Reviewer comment (Experiments):
Data is also a concern here. While it's reasonable to leverage more data, it introduces an element of unfairness because the data for the compared models is not consistent. This makes it challenging to draw valid conclusions, and I believe this aspect may also impact the overall findings.

Author response:
**Answer**: Thank you for your suggestion. Comparing with other methods under the same training data is not realistic because many works use their own private datasets, such as CLIP, Florence, DALLE, etc. 

Since jointly learning cross-modal contrastive optimization, text-to-image generation, and image-to-text generation is a new setting, ours serves as an initial exploration of how those three losses impact each other, and which model structure is more effective and efficient. On those two core problems, all our experiments and ablations are **conducted fairly with the same amount of data**. So the findings of our paper should be valid and useful to the community.

Reviewer comment (Reproducibility):
The optimization process by the authors appears rather intricate, involving the design of numerous loss functions and hyperparameters. This complexity in model design makes replication challenging because the network and parameters are closely tied to the specific dataset they chose. I believe this is also a weakness of the study.

Author response:
**Answer**: Thanks for pointing out. It’s common that multi-task learning and multi-objective optimization usually involve more hyperparameters due to multiple losses. We have listed all the hyperparameters in Sec. 4.1 and Tab. 7 of the original submission for a clear reference and also ablated the loss coefficients in Sec. 6.2 (Tab.8).

Reviewer comment (Experiments):
In the Model Initialization section, the authors mention using CoCa but do not provide a specific explanation for this choice. They do not clarify why they didn't opt for pre-trained BERT models, sentence transformers, or CLIP-based sentence encoders. It would be beneficial if the authors could offer more insight into the rationale behind their selection of COCA as the initialization method and why they didn't consider other pre-trained models for this purpose.

Author response:
**Answer**: It’s common in previous works that a pre-trained text encoder is used for image generation(DALLE, Parti, SD, etc). We expect that the performance of CLIP and COCA should be similar because they both encode text in a cross-modality way. Due to computation resources and limited time, we didn’t further explore using CLIP to replace Coca. But more essentially, **we did ablate initializing from coca vs training from scratch in Tab.6 and found the training from scratch can bring comparable performance.**

Reviewer comment (Experiments):
In Fig. 2, the fact that ViT-VGGAN is frozen to some extent can be considered a weakness, as the existing CoBIT framework assumes the availability of a well-pretrained ViT-VQGAN. This assumption is partially valid, but if the domain changes, it implies that the entire model would need to be re-pretrained. Moreover, if there's a new VQGAN model, the entire model would also require retraining. I am curious about how the authors address this issue, as this paper primarily focuses on pre-training and should ideally rely on existing model checkpoints as much as possible.

Author response:
**Answer**: Good point. Freezing VQGAN/VQVAE in image generation model is a common operation not only in auto-regressive methods (DALL-E, Parti), but also in diffusion models (SD). In Ernie-ViLG [1], they try to unfreeze the VQGAN, and in large-scale training, it causes instability in training and the model can’t converge.

[1] Zhang, Han, et al. "Ernie-vilg: Unified generative pre-training for bidirectional vision-language generation." arXiv preprint arXiv:2112.15283 (2021).

Reviewer comment (Experiments):
The results of the Linear Probing experiment show a significant improvement, yet the authors have not provided a more substantial explanation. If Linear Probing is the reason for the 1% improvement in performance, how would fine-tuning more parameters affect the outcome?

Author response:
**Answer**: Linear probing can usually improve upon zeroshot in general not only in our experiment but also in previous works (CLIP, ALIGN, etc). We think the reason why linear probing can improve the performance over zero-shot performance is that it learns a single-layer adaptation to target categories from the pre-trained knowledge. As for finetuning more parameters, similar to previous works (CLIP, ALIGN, etc), we expect our model to gradually improve until saturated.

Reviewer comment (Experiments):
The authors trained image captioning using only cross-entropy loss and overlooked RL-based rewards, which are an important component. The authors should have at least discussed how to integrate RL-based methods into this model because RL-based methods, such as self-critical sequence training (SCST), also impose specific requirements on model design.

Author response:
**Answer**: Thank you for your suggestion. SCST [2] is usually used to optimize the image captioning model with CIDEr metrics. When applying SCST to CoBIT, no specific model structure change is needed. To be more specific, CoBIT will conduct two feedforwards: one is sampling and the other is greedy inference. CIDEr score will be computed between each of the generated sentences and GT sentence. Then the REINFROCE algorithm is applied, which regards the greedy estimated sentence’s CIDEr score as the baseline and the sampled sentence as the prediction. Therefore, samples from the model that return a higher reward than the greedy estimated sentence will be encouraged, or increased in probability, while samples that result in a lower reward will be suppressed. For the sake of limited budget for experiments and the simplicity of our framework (since our CoBIT already has multiple objectives), we didn’t incorporate this in our experiment but we will consider adding it in the final version.

[2]. Rennie, Steven J., et al. "Self-critical sequence training for image captioning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Reviewer comment (Evaluation):
Furthermore, when reporting captioning results, the authors should present them on the MSCOCO online test set, as it provides a fair evaluation benchmark, rather than reporting them on a local set.

Author response:
**Answer**: The reason we report MSCOCO captioning result in Karpathy-split test set is that it’s commonly used in previous works(CoCa, BEIT, PALI, SimVLM, OFA, BLIP1/2, etc) for a more comprehensive comparison. We fairly compare with previous works by making sure no test data leakage in training. The reason for not presenting in MSCOCO online test set is that only very few previous works reported their result on this set (in recent works about image-text foundation models, we only find BUTD(2017), VinVL(2021) and GIT(2022) have official results reported), which makes a comprehensive comparison difficult.