Paper: COSA: Concatenated Sample Pretrained Vision-Language Foundation Model (ICLR-2024)

Reviewer comment (Novelty):
Technical contributions: The proposed method is simple and effective. However, the proposed method is not surprising enough, because using pictures to enhance video pre-training has been quite explored in the field of CV/ vision-language pre-training field. This paper combines many existing pre-training methods, so the technology sharing is limited.

Author response:
## **Q1:  Using pictures to enhance video pre-training has been quite explored in the field of CV/ vision-language pre-training field**  
## **A1:** 
 Indeed many video-language pretraining methods also using image-text data (CC3M) as their training corpora, **but they view each picture as individual training sample.** By contrast, **COSA is the first work to utilize pictures to compose pesudo video-paragraph training data and proved that can  further improve performances of series of understanding and generation tasks**. In other words, we believe that our technical contribution lies in COSA's proposal of a novel approach to effectively utilize image data for video-language pretraining, rather than just employing both image and video data in pretraining, which has already been widely explored.

Reviewer comment (Theory):
More explanation: The continuous frames in the video are similar, and there are relations between different events. Establishing event-level correlation includes two parts: (1) the first part is to distinguish between different events. (2) the second part is to make temporal inferences between similar or related frames. The proposed method randomly splices several pictures, but there is no correlation between these pictures, so the model can only distinguish different events, but can not make the model time sequence inference between frames. Therefore, I do not think that the proposed method fully corresponds to its motivations.

Author response:
## **Q2: Proposed method does not fully correspond to its motivations.**
## **A2:**
 We also agree on your analysis of two parts of temporal modeling, i.e., inter-scene temporal correlation and intra-scene temporal correlations. However, **as wrote in the "Temporal Learning in Video-Language Pretraining (VidLP)" section of Related Work** in the paper, we have attempted to let **"event-level temporal" refer explicitly to the first part, i.e. inter-scene temporal, instead of both parts**, and we use "action-level temporal" to refer to the second part, i.e. intra-scene temporal.  There are some works like OmniVL and HiTeA tried to model action-level temporal during pretraining, while COSA emphasizes more at event-level temporal learning, considering that action-level (intra-scene) needs consecutive images in a scene, which is hard to realize using pure image-text data. We also think that strengthening both action-level and event-level temporal modeling in a unified framework could be a meaningful future video-language research direction. **The motivation of current version of COSA  is to strength "event-level" (first part) temporal modeling only instead of both "action-level" (second part) and "event-level" temporal modeling.** **And we have made the illustration more clear in the updated version of Related Work**.

Reviewer comment (Evaluation):
In Table 9, why is it better to concatenate random images for training than to concatenate only semantically similar images?

Author response:
## **Q3: Why is it better to concatenate random images for training than to concatenate only semantically similar images**
## **A3:**
In our experiments, COSA trained with random sampling works better than the one trained with relevant sampling. **We make visualizations of different sampling methods and add explanations in the updated version of supplementary materials.** 

Specifically, as figure 1 shown in the supplementary material, when we observe a real video-language sample from ActivityNet caption dataset (the top green part of figure), we can find that the 1) sampled vision frames are relevant and coherent, and 2) the corresponding texts are very informative and can strictly match to different frames in temporal order. 

Motivated by this, an ideal  large-scale  pretraining dataset for video-language tasks should also possess those two characteristics to help model learn the capabilities of perceiving events happend in videos and their relative order.  In this work, we propose COSA to imitate those pretrain dataset from image-text data. And the examples of COSA with different sampling methods are shown in the bottom pink part of the figure, from which we can find that even though the random sampling example shows little relevance between four independent scenes, but it has the strength that each sentance can strictly correspond to one image, which satisfies the second charatersic. 

By contrast, even though relevant (vision/text similarity) sampling examples have shown vision relevance  to some extent, they have strong semantic redundancy, and one sentence can correspond to many frames, which could confuse models' event-level temporal learning. And we think that's why it performs weaker than random sampling. In addition, considering that the scene changes are somewhat hard in random sampling, we think other soft sampling methods could be a future research direction.

Reviewer comment (Reproducibility):
It is better to give the weights of the 6 losses (training objectives) and the size of the input image in the implementation details.

Author response:
## **Q4: It is better to give the weights of the 6 losses (training objectives) and the size of the input image in the implementation details.**
## **A4:** 
We haven't intended to tune the 6 training losses, and they  are simply equally weighted. With regards to input image size, as presented in Table 3 of Appendix, for pretraining and video tasks finetuning, we use 224 resolution, and for image-text tasks finetuning, we use 384 for QA and retrieval, and 480 for captioning. **We have added loss weights and input image size to the implementation details at the updated version of paper, thanks for your advice**.

Reviewer comment (Novelty):
The method was more like a trick of data augmentation instead of a significant technical contribution, as it just simply concatenated images and their corresponding captions and it was not very surprising to observe performance improvements.

Author response:
- We think viewing COSA from the data augmentation perspective also makes sense, considering that the core operation of COSA is to concatenate image-text data into pseudo video-paragraph data, and learn event-level temporal correspondences which are needed for video-text downstream tasks. **However, as far as we know, we are the first to compose pseudo video-paragraph data from image-data, demonstrating its effectiveness across abundant both image-text and video-text tasks, making detailed comparison with the common SST baseline, and verifying its generality across different dataset and model scales, so we think COSA has its novelty.** 

- In addition, as suggested, we add experiments to make comparisons with other data augmentation method. Mixup is first proposed for image classification tasks which sample two images and interpolate both raw images and their labels, in equal weight. Motivated by Mixup, Mixgen promotes it from image field to image-text field, replacing labels with captions and keeping image processing remained (average raw pixels of two images). We add the comparison results as follows, in which we find that compared to SST baseline, mixgen severely decreases the performance of downstream video-text tasks, while COSA improves SST baseline evidently. We attribute this phenomenon to that Mixgen simply averages different images into one single image without any temporal modaling, leaving event-level temporal learning unconsidered, thus the more images averaged (4 vs 2), the more performance degrades. By contrast, COSA concatenates multiple images  into videos and encodes temporal information via temporal position embeddings, which can effectively learn event-level temporal correspondence, and more images can further improve performances. These results demonstrate that COSA is more general and effective compared to Mixgen. 
| Method | # Concatenated or mixed Images | DiDeMo-RET (R@1) | ActivityNet-RET (R@1) | VIST-CAP(CIDEr) | TVC-CAP(CIDEr) | TGIF-QA(Acc) |
| --- | --- | --- | --- | --- | --- | --- |
| SST | - | 49.2 | 43.8 | 14.8 | 53.9 | 73.0 |
| COSA(Ours) | 2 | 49.1 | 45.5 | 15.3 | 54.5 | 73.5 |
| COSA(Ours) | 4 | 54.8 | 51.2 | 20.7 | 55.4 | 73.7 |
| Mixgen | 2 | 46.5 | 42.9 | 11.9 | 53.6 | 73.1 |
| Mixgen | 4 | 40.8 | 37.2 | 10.0 | 52.9 | 72.3 |

Reviewer comment (Experiments):
As it was mentioned in the paper that apart from modified objectives, COSA also included original objectives for pre-training on image-text pairs. It was a complicated design to have so many training objectives and it was unclear how they were weighted (seemed to be equally weighted). Even though there was an ablation study of training objectives in Table 7, it still did not explain well the contributions of each item.

Author response:
Since there are up to six losses (ITC/ITM/CITC/CITM/CMLM/CGM) used in COSA, we do not tune their weights considering the complexity. Using identical weights already achieves good performance. We also make experiments to analyze the function of each loss. For clarity, we group ITC and ITM into L_align (L_align=L_itc+L_itm), and group MLM and GM into L_mask (L_mask = L_mlm+L_gm). The analysis is as follows:
- ITC improves retrieval task (line1 vs line0), and ITM make further improvements via fine-grained multimodal fusion and rereanking (line2 vs line1).
 - MLM (bi-direction self-attention masks) can improve caption and QA tasks (line3 vs line0), and adding GM (causal self-attention mask) can further enhance performance (line4 vs line3).
-  Using L_align and L_mask together leads to notable improvements on all tasks (line5 vs line4).
-  Regarding the 'Concatenate' operation in COSA, replacing L_mask with L_Cmask and incorporating L_Calign both contribute to improved performance across all tasks (line6 vs line5, line7 vs line6).

| Line | Losses  | DiDeMo-RET | MSRVTT-RET | TVC-CAP | TGIF_QA |
| --- | --- | --- | --- | --- | --- |
| 0 | - | 26.9 | 34.1 | 51.0 | 69.4 |
| 1 | L_itc | 38.4 | 37.1 | 49.7 | 66.6 |
| 2 | L_align (L_itc+L_itm) | 48.4 | 38.3 | 50.3 | 69.2 |
| 3 | L_mlm | 26.0 | 31.6 | 52.1 | 72.1 |
| 4 | L_mask (L_mlm+L_gm)  | 30.9 | 33.5 | 53.3 | 72.9 |
| 5 | L_align+L_mask | 49.2 | 39.4 | 53.9 | 73.0 |
| 6 | L_align+L_Cmask | 53.0 | 42.8  | 55.7  | 73.7 |
| 7 | L_align+L_Calign+L_Cmask | 54.8  | 43.5  | 55.4 | 73.7 |

Reviewer comment (Experiments):
The method leveraged the average pooled [CLS] token for each image as the final representation for the pseudo video. In this way, there was actually no temporal information considered. And the selected downstream tasks were less dependent on temporal information in the meanwhile. It would be better if tasks such as temporal action localization were included to show whether COSA can improve those tasks. In addition, since temporal information did not play any role in current method, I am afraid that using augmentations like mixup for videos/images might lead to similar performance gain, as shown in [1].

Author response:
- We evaluate COSA and SST on **temporal action localization (TAL)** task. Specifically, we take **actionformer** as baseline methods and evaluate on **ActivityNet dataset** with features extracted from vision encoders of COSA-B and SST-B respectively. It is noted that they are both trained with the same datasets and iterations. The results are shown in the following table, from which we can find that both SST and COSA improve performance over baseline. Furthermore, COSA surpasses SST, demonstrating its efficacy on the localization task.
| Method | Vision feature | mAP |
| --- | --- | --- |
| Actionformer | Swin-B | 33.40 |
| Actionformer | SST-B | 34.76 |
| Actionformer | COSA-B | 35.10 |

- However, the improvements appear limited, which we believe may be due to the fact that COSA is pre-trained as an integrated vision-language framework, but when fine-tuned on TAL, only its vision encoder is utilized. This approach might not fully leverage the strengths of the COSA framework. So we additionally evaluate COSA on **text-guided video grounding task** (i.e., moment retrieval). 
Specifically, inspired by **MMN**, we sparsely sample 16 frames per video and forward them to the vision encoder,  getting a 1d feature map whose size equals to 16\*C (C is the hidden size of channel). After that we transform the 1d feature map to 2d map whose size is 16\*16*C. Each feature located at (i, j) in the feature map represents a pre-defined anchor with fixed start and end timestamps (which are the timestamps of i-th and j-th frames), and they are computed by averaging the features of i-th and j-th frames along channel. It is noted that only the upper triangle part of 2d map is valid due to that start time must be smaller than end time. Input querys are processed by the text encoder of COSA.  The whole training losses consist of contrastive loss and IOU loss, following MMN (constrastive head is inherited from pretrained COSA model and IOU head is newly initialized). All parameters of networks are updated during finetuing. We conducted experiments on the **Charades dataset**. As presented in the following table, COSA-B surpasses SST-B with large margins on the R@1 metric of different IOU scores, strongly proving that COSA can also benefit localization tasks.
| Method | R@1_IOU0.3 | R@1_IOU0.5 | R@1_IOU0.7 |
| --- | --- | --- | --- |
| Baseline(w/o pretrain) | 67.69 | 54.49 | 32.15 |
| SST-B | 68.66 | 56.45 | 34.84 |
| COSA-B | 69.62 | 58.01 | 36.40 |

Reviewer comment (Experiments):
Is it possible for the authors to include tasks which rely much on temporal information like temporal action localization? This would provide better understanding of the proposed method.

Author response:
- We evaluate COSA and SST on **temporal action localization (TAL)** task. Specifically, we take **actionformer** as baseline methods and evaluate on **ActivityNet dataset** with features extracted from vision encoders of COSA-B and SST-B respectively. It is noted that they are both trained with the same datasets and iterations. The results are shown in the following table, from which we can find that both SST and COSA improve performance over baseline. Furthermore, COSA surpasses SST, demonstrating its efficacy on the localization task.
| Method | Vision feature | mAP |
| --- | --- | --- |
| Actionformer | Swin-B | 33.40 |
| Actionformer | SST-B | 34.76 |
| Actionformer | COSA-B | 35.10 |

- However, the improvements appear limited, which we believe may be due to the fact that COSA is pre-trained as an integrated vision-language framework, but when fine-tuned on TAL, only its vision encoder is utilized. This approach might not fully leverage the strengths of the COSA framework. So we additionally evaluate COSA on **text-guided video grounding task** (i.e., moment retrieval). 
Specifically, inspired by **MMN**, we sparsely sample 16 frames per video and forward them to the vision encoder,  getting a 1d feature map whose size equals to 16\*C (C is the hidden size of channel). After that we transform the 1d feature map to 2d map whose size is 16\*16*C. Each feature located at (i, j) in the feature map represents a pre-defined anchor with fixed start and end timestamps (which are the timestamps of i-th and j-th frames), and they are computed by averaging the features of i-th and j-th frames along channel. It is noted that only the upper triangle part of 2d map is valid due to that start time must be smaller than end time. Input querys are processed by the text encoder of COSA.  The whole training losses consist of contrastive loss and IOU loss, following MMN (constrastive head is inherited from pretrained COSA model and IOU head is newly initialized). All parameters of networks are updated during finetuing. We conducted experiments on the **Charades dataset**. As presented in the following table, COSA-B surpasses SST-B with large margins on the R@1 metric of different IOU scores, strongly proving that COSA can also benefit localization tasks.
| Method | R@1_IOU0.3 | R@1_IOU0.5 | R@1_IOU0.7 |
| --- | --- | --- | --- |
| Baseline(w/o pretrain) | 67.69 | 54.49 | 32.15 |
| SST-B | 68.66 | 56.45 | 34.84 |
| COSA-B | 69.62 | 58.01 | 36.40 |

Reviewer comment (Experiments):
It would be better if results with different initializations can be presented to remove my concern about better CLIP initialization.

Author response:
- In fact, for fair comparison with state-of-the-art methods, we train different scales of COSA model with different vision encoders and training corpus. As depicted in the following table (the same as Table 1 in the paper), COSA-B takes **Swin-Base** trained on ImageNet as vision encoder, **instead of CLIP** pretrained on 400M WebDataset.
| Model |  Vision Encoder |
| --- | --- |
| COSA-B | Swin-B |
| COSA-L | CLIP/ViT-L/14 |
| COSA |  EVAClip/VIT-G/14 |

- **We copy the comparison results of COSA-B from Table 2 in the paper below, and additionally add a vision encoder column**. It is noted that **all methods use vision backbones pretrained on ImageNet, and thus we believe a fair comparison is warranted.** Methods using Swin as backbone generally outperform the ones using ViT as backbone, and COSA surpasses other methods like Clover, VIOLETv2, LF-VILA, which also use Swin. Considering that Swin may be marginlly better than ViT backbone, we are training a COSA-B(ViT) model for a absolute fair comparison with methods initialized with ViT, such as MILES. However, this experiment requires a longer training period compared to our other experiments conducted for this rebuttal, due to the increased number of training iterations and the non-fixed vision backbone. Consequently, we are unable to complete it before the end of the rebuttal period. We will add this comparison results at the final version of paper.
| Method | Vision encoder | Example | MSRVTT | DiDeMo | LSMDC | ActivityNet |
| --- | --- | --- | --- | --- | --- | --- |
| ClipBert | ResNet50 | 5.4M | 22.0 /46.8/59.9  | 20.4/48.0 /60.8 | - | 21.3/49.0/63.5 |
| Frozen | ViT-B | 5M | 31.0/59.5/70.5  | 34.6/65.0/74.7 | 15.0/30.8 /39.8 | - |
| BridgeFormer | ViT-B | 5M | 37.6/64.8/75.1 | 37.0/62.2/73.9 | 17.9/35.4/44.5 | - |
| MILES | ViT-B | 5M | 37.7/63.6/73.8 | 36.6/63.9/74.0  | 17.8/35.6/44.1  | - |
| OA-Trans | ViT-B | 5M | 35.8/63.4/76.5 | 34.8/64.4/75.1 | 18.2/34.3/43.7 | - |
| Clover | Swin-B | 5M | 38.6/67.4/76.4 | 45.1/74.3/82.2 | 22.7/42.0/52.6 | - |
| VIOLETv2 | Swin-B  | 8.5M | 37.2/64.8/75.8  | 47.9/76.5/84.1  | 24.0/43.5/54.1 | - |
| LF-VILA | Swin-B | 5M |-  | 35.0/64.5/75.8 | - | 35.3/65.4/- |
| COSA-B | Swin-B | 5M | **42.2/69.0/79.0** | **57.8/80.6/87.9** | **27.3/46.7/55.2** | **55.6/80.7/89.1**  |

Reviewer comment (Experiments):
It was worth trying data augmentations like mixup and it might lead to similar performance gain as demonstrated in the paper.

Author response:
- We think viewing COSA from the data augmentation perspective also makes sense, considering that the core operation of COSA is to concatenate image-text data into pseudo video-paragraph data, and learn event-level temporal correspondences which are needed for video-text downstream tasks. **However, as far as we know, we are the first to compose pseudo video-paragraph data from image-data, demonstrating its effectiveness across abundant both image-text and video-text tasks, making detailed comparison with the common SST baseline, and verifying its generality across different dataset and model scales, so we think COSA has its novelty.** 

- In addition, as suggested, we add experiments to make comparisons with other data augmentation method. Mixup is first proposed for image classification tasks which sample two images and interpolate both raw images and their labels, in equal weight. Motivated by Mixup, Mixgen promotes it from image field to image-text field, replacing labels with captions and keeping image processing remained (average raw pixels of two images). We add the comparison results as follows, in which we find that compared to SST baseline, mixgen severely decreases the performance of downstream video-text tasks, while COSA improves SST baseline evidently. We attribute this phenomenon to that Mixgen simply averages different images into one single image without any temporal modaling, leaving event-level temporal learning unconsidered, thus the more images averaged (4 vs 2), the more performance degrades. By contrast, COSA concatenates multiple images  into videos and encodes temporal information via temporal position embeddings, which can effectively learn event-level temporal correspondence, and more images can further improve performances. These results demonstrate that COSA is more general and effective compared to Mixgen. 
| Method | # Concatenated or mixed Images | DiDeMo-RET (R@1) | ActivityNet-RET (R@1) | VIST-CAP(CIDEr) | TVC-CAP(CIDEr) | TGIF-QA(Acc) |
| --- | --- | --- | --- | --- | --- | --- |
| SST | - | 49.2 | 43.8 | 14.8 | 53.9 | 73.0 |
| COSA(Ours) | 2 | 49.1 | 45.5 | 15.3 | 54.5 | 73.5 |
| COSA(Ours) | 4 | 54.8 | 51.2 | 20.7 | 55.4 | 73.7 |
| Mixgen | 2 | 46.5 | 42.9 | 11.9 | 53.6 | 73.1 |
| Mixgen | 4 | 40.8 | 37.2 | 10.0 | 52.9 | 72.3 |

Reviewer comment (Presentation):
The data shown in Table 1 is confusing. The data for COSA-L is 417M, while the data volume for COSA is 415M.

Author response:
## **Q3: Training data volume of COSA-L and COSA is confusing.**
## **A3:**
Sorry for the confusion caused. For clarity, the details of the data used in training COSA/COSA-L are listed in the table below. The difference in data volume is due to the **exclusion of the Webvid-2.5M dataset in the COSA training**. The reason of this exclusion is explained in our response (A2) to "Q2: Necessity of including the video dataset (Web2Vid)". As noted in the caption of Table 1, since the LAION-102M used in COSA is sampled from the training corpus of the EVAClip vision backbone (LAION-400M), we count them collectively as 400M.

| Model | Vision-text pairs utilized in vision-language pretraining | Vision-text pairs utilized in vision encoder pretraining | Total |
| --- | --- | --- | --- |
| COSA-L | CC14M (**15M**) + Webvid2.5M(**2.2M**)  | Webdata400M(**400M**) | **417M** |
| COSA | CC14M (**15M**) + LAION102M(102M, overlapped with LAION400M) | LAION400M(**400M**) | **415M** |

- "CC14M" combines the CC3M, CC12M, COCO, VG, and SBU caption datasets, and after the removal of invalid links, totaling 15M  pairs remain accessible.

- "Webvid2.5M" initially contains 2.5M vision-text pairs, and after filtering invalid links, we achieve 2.2M pairs for training.

Reviewer comment (Experiments):
The results in Table 7 and Table 7 are also confusing. The best performance is based on 6 pretraining task? Which pre-training tasks were used in the overall experimental results of COSA? Is the WebVid2.5M dataset more important，the results for COSA 4 frames?

Author response:
## **Q4:  The results in Table 7 and Table 8 are also confusing. The best performance is based on 6 pretraining task? Which pre-training tasks were used in the overall experimental results of COSA? Is the WebVid2.5M dataset more important, the results for COSA 4 frames?**
## **A4:**
- Yes.  **As written in Table 7 and "Training Objectives" in Section 4.3**, the best performance is based on 6 pretraining tasks, and they  are also used in all other COSA experiment tables.
- **Table 8** targets at demonstrating COSA's generality to different datasets and backbones. We want to **emphasize the comparison between methods (i.e. SST baseline and COSA), instead of datasets (i.e., cc3m and webvid)**. **In addition, we compare webvid and cc3m in the table below, and conduct an additional experiments that use both dataset together**. From the results we can find that separately using one dataset can achieve similar performance, and using them together further improves performance on all tasks.

| Method | Enc  | Dataset | Frame | DiD(R) | ANET(R) | MSR(R) |  YOU(C) | MSR(Q) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| COSA  | CLIP-B | WebVid2.5M | 1 | 54.7 | 52.2 |  44.0 | 93.4 | 46.4 |
| COSA  | CLIP-B | CC3M | 1 | 54.8 | 51.2 | 43.5 | 91.9 | 46.5 |
| COSA  | CLIP-B | WebVid2.5M+CC3M | 1 | 55.8 | 54.1 | 46.2 | 100.5 | 47.0 |

Reviewer comment (Evaluation):
It makes sense that pseudo video-paragraph data in pre-training can mitigate the gap between pre-training and fine-tuning in image-text pertaining. However, intuitively, the discontinuity of semantics in pseudo video-paragraph data should hurt compared with relevant video-paragraph data because in downstream videos, image and text are indeed relevant. But in Tab9, it seems random sampling is better than relevant sampling, which is kind of counter-intuitive. Can the authors explain more about it?

Author response:
In our experiments, COSA trained with random sampling works better than the one trained with relevant sampling. **We make visualizations of different sampling methods and add explanations in the Figure 1 of updated version of supplementary materials**. 

Specifically, as Figure 1 shown in the supplementary material, when we observe a real video-language sample from ActivityNet caption dataset (the top green part of figure), we can find that the 1) sampled vision frames are relevant and coherent, and 2) the corresponding texts are very informative and can strictly match to different frames in temporal order. 

Motivated by this, an ideal  large-scale  pretraining dataset for video-language tasks should also possess those two characteristics to help model learn the capabilities of perceiving events happened in videos and their relative order.  In this work, we propose COSA to imitate those pretrain dataset from image-text data. And the examples of COSA with different sampling methods are shown in the bottom pink part of the figure, from which we can find that even though the random sampling example shows little relevance between four independent scenes, but it has the strength that each sentence can strictly correspond to one image, which satisfies the second characteristic. 

By contrast, even though relevant (vision/text similarity) sampling examples have shown vision relevance  to some extent, they have strong semantic redundancy, and one sentence can correspond to many frames, which could confuse models' event-level temporal learning. We believe that's why it performs worse than random sampling. In addition, considering that the scene changes are somewhat abrupt in random sampling, we think smoother sampling methods could be a future research direction.

Reviewer comment (Experiments):
When having seen the same number of samples, whether COSA is better than SST in `image-text downstream tasks`? Basically, I want to see the comparison like Figure 4 in image-text downstream tasks. I am okay with this observation not holding anymore in image-text downstream tasks because essentially video-paragraph and image-text are different domains.

Author response:
In fact, **Figure 4 already contains two image tasks**, including MSCOCO-RET (text-to-image retrieval) and VIST-CAP (image story telling). We here represent the comparison on those two benchmarks  in table below.


| Method    | Iteration | MSCOCO-RET (Recall@1) | VIST-CAP (Cider) |
|-----------|-----------|-----------------------|------------------|
| SST-30K   | 30K       | 53.9                  | 14.8             |
| **COSA-30K**  | 30K       | 54.7                  | 20.7             |
| SST-60K   | 60K       | 54.8                  | 18.6             |
| COSA-60K  | 60K       | 55.3                  | 24.1             |
| **SST-120K**  | 120K      | 54.6                  | 19.1             |
| COSA-120K | 120K      | 55.8                  | 26.0             |

From the table we can find that when comparing COSA and SST under the same examples seen setting (COSA-30K vs SST-120K), COSA-30K achieves 54.7 R@1 and 20.7 CIDEr, which outperforms SST-120K who achieves 54.6 R@1 and 19.1 CIDEr. This demonstrates that COSA is both more effective and efficient than SST (COSA-30K only takes less than half the training time of SST-120K), for both image-text and video-text tasks, showing its generality as a vision-language foundation model. It is noted that when compared under the same iteration setting (both methods are converged), COSA-120K outperforms SST-120K by a significant margin.

Reviewer comment (Experiments):
I want to see how this method performs in zero-shot image-text tasks. Considering the domain gap, I suspect it might perform worse than some image-text pre-trained methods that COSA can outperform when finetuning.

Author response:
Considering MSCOCO is included in CC4M dataset which has been used  in the pretraining process of COSA, we test COSA on two image-text zero-shot benchmarks including **Flickr-30K image-text retrieval** and **Nocaps image caption**, and compare COSA with other pretraining methods.  
For Flickr-30K retrieval, we test in two settings: 
- direct zero-shot evaluation. From the results we can find that COSA outperforms methods including CLIP/ALIGN/FILIP/Florence/CoCa with evident strengths. Considering that compared methods are all two-tower models while COSA use ITM to rerank, we test COSA-ITC that removes the ITM refining process, we can find that COSA-ITC also achieves better text-to-image and decent image-to-text zero-shot performances.
| Method   | Finetune on MSCOCO | Text-to-Image R@1/R@5/R@10 | Image-to-Text R@1/R@5/R@10 |
|----------|--------------------|----------------------------|----------------------------|
| CLIP     | N                  |   68.7 90.6 95.2           | 88.0 98.7 99.4             |
| ALIGN    | N                  | 75.7/93.8/96.8             | 88.6/98.7/99.7             |
| FILIP    | N                  |  75.0 93.4 96.3            | 89.8 99.2 99.8             |
| Florence | N                  | 76.7/93.6/-                | 90.9/99.1/-                |
| CoCa     | N                  | 80.4/95.7/97.7             | 92.5/99.5/99.9             |
| COSA-ITC | N                  | 84.4/**97.2**/**98.5**            | 88.0/99.7/99.9             |
| COSA     | N                  | **87.2**/97.0/97.9             | **96.8**/**100.0**/**100.0**           |

- finetuning on MSCOCO retrieval dataset and then tested on Flickr30K. In this setting, all compared methods use ITM reranking and have an intermediate finetuning process on MSCOCO dataset. COSA surpasses ALBEF/BLIP/mPLUG-2/BLIP-2 with evident margins.
| Method | Finetune on MSCOCO | Text-to-Image R@1/R@5/R@10 | Image-to-Text R@1/R@5/R@10 |
| --- | --- | --- | --- |
| ALBEF | Y | 85.6/97.5/98.9 | 95.9 99.8 100.0 |
| BLIP  | Y | 86.7/97.3/98.7  | 96.7/100.0/100.0 |
| mPLUG-2  | Y | 88.1/97.6/99.1 | 97.2 100.0 100.0 |
| BLIP-2 | Y | 89.7/98.1/98.9 | 97.6/100.0/100.0 |
| COSA (Ours) | Y | **90.2**/**98.4**/**99.4**  | **98.3**/**100.0**/**100.0**  |

For the Nocaps zero-shot caption dataset, it is noted that there are models with LLM as a multimodal decoder achieving higher performance, such as BLIP-2. Considering that COSA utilizes BERT-base as its multimodal decoder, we only compare it with similar models. Specifically, we find that COSA outperforms OSCAR and VinVL by large margins and achieves comparable performance with SimVLM and BLIP.
|  | Use LLM | Cider | Spice |
| --- | --- | --- | --- |
| BLIP-2 | Y | 119.7  | 15.4 |
| OSCAR | N | 80.9  | 11.3  |
| VinVL | N | 95.5  | 13.5 |
| BLIP | N |  **113.2**  | 14.8 |
| SimVLM | N |  112.2 | - |
| COSA(Ours) | N | 113.0 | **15.0** |