Paper: CrossTVR: Multi-Grained Re-Ranker for Text Video Retrieval with Frozen Image Encoders (ICLR-2024)

Reviewer comment (Evaluation):
It seems that the experimental results do not include retrieval performance (e.g. stage 1 / stage 2/ pre- and post-processing), which is important in terms of a cross-modal retrieval topic.

Author response:
### 1. Experimental results in stage1/stage2/post-processing.

The results of our searches are in Tables 1-5. The original cosine similarity method 'CLIP-VIP' is the stage1 result.  The cosine similarity method togher with our method  'CLIP-VIP + Ours' is the stage 2 result. The method denoted with (*) is the post-processing result. For better presentation, we restate  in the following table the results of the stage1/stage2/post-processing in the MSRVTT dataset.

|                                 | R@1  | R@5  | MnR  |
|---------------------------------|------|------|------|
| CLIP-VIP (stage1)                   | 50.1 | 74.8 | -    |
| CLIP-VIP+Ours  (stage2)         | 51.9 | 76.8 | 14.3 |
| CLIP-VIP+Ours*(post-processing) | 56.9 | 80.0 | 11.5 |
| TS2-Net (stage1)                      | 47.0 | 74.5 | 13.0  |
| TS2-Net+Ours  (stage2)         | 50.0 | 75.7 | 12.0 |
| TS2-Net+Ours*(post-processing) | 52.8 | 78.9 | 11.2 |

Reviewer comment (Experiments):
Again, in terms of a retrieval task, it is important to verify the effectiveness under a huge database setting. However, the most large dataset only contains 118,081 videos which is much less than industiral scales such as YouTube. At least, more distractors should be included if scaling positive text-video pairs is a concern.

Author response:
### 2.Larger datasets and scaling video-text pairs. More distractors should be included if scaling positive text-video pairs is a concern.

Yes, we do not have such a industrial scale video retrieval dataset like YouTube.  We create a larger validation dataset by combining the vallidation set of MSRVTT, LSMDC, and Didemo dataset.  LSMDC datasets contains 118,081 videos  is currently the largest video-text retrieval public dataset for scholarly.  And MSVTT datasets contains 20 titles per video, with up to 200,000 video text pairs. Didemo contains more than 10,000 videos and 40,000 captions, 
We use the positive pairs from MSRVTT as groundtruth and use validatiaon set from LSMDC and Didemo as distractors . We use the model trained on MSRVTT dataset for evaluation. The following results show that our method is still effective in large scale retrieval dataset.

|                                 | R@1  | R@5  | MnR  |
|---------------------------------|------|------|------|
| TS2-Net (stage1)               |  37.2   |63.5   | 44.1  |
| TS2-Net+Ours  (stage2)         | 40.5 | 67.8   | 43.8 |
| TS2-Net+Ours*(post-processing) | 42.7 | 68.0 | 47.0 |

Reviewer comment (Presentation):
I think the visualization is not enough. For example, the retrieval results of different work or the effect of proposed re-ranking could be included as supplementary materials if space limitation is a concern. Besides, the visualization should verify the effectiveness of proposed saptial text attention and temporal text attention.

Author response:
### 3.More visualization results about re-ranking and spatial text attention and temporal text  attention.

We add more text to video re-ranking visualization results in Figure.7 and video to text re-ranking results in Figure. 8 of the supplementary material, and visualizations of spatial attention and temporal attention are presented in Figure 5 of the paper. Spatial attention allows us to identify small objects at the frame level, while temporal attention helps us to discriminate continuous actions at the video level.

Reviewer comment (Reproducibility):
The token selector is employed as a trade-off between missing detailed information and bringing redundant information issues. The implementation of such a module is a MLP followed by a Softmax Layer, which predicts each token's importance score and selects the M most informative tokens as output. The question is what is the ground truth (GT) for such predictions? It seems that it is even impossible for human beings to label. Note that this is not a single label classfication problem.

Author response:
### 4. The ground truth of Token selector predicted importance score.

Yes, the token selector aim to predict the most sailent video level features.  As we do not have the groundtruth of importance score, the prediction may be not accurate. However, in frame-level cross attention, we use all tokens to make sure all details information are not missed. This is the reason why combine video level cross attention and frame level cross attention can further improves the performance.

Reviewer comment (Reproducibility):
Explain the specific implementation and mechanism of video-level and frame-level cross attention. And how do these components operate and interact.

Author response:
Please refer to the supplementary material Fig. 5, which shows the details of our video-level and frame-level cross attention.  Specifically, for frame text attention, learnable queries are first concatenated with text tokens. These queries interact through self attention layer and then interact with each frame features in cross attention layer. These queries feed into video text attention after averaging. For video text attention, these queries interact with the selected video token in cross attention.

Reviewer comment (Experiments):
Although frozen vision encoder reduce computational costs, this approach may limit the model's flexibility and ability to adapt to different tasks.

Author response:
Yes, we have not try other video understanding tasks like video QA, video caption so we do not know whether the frozen vison encoder strategy is also applicable on these tasks. However,  we are certain that finetuning the whole vision encoder will not perform worse than the frozen vision encoder. Therefore, when adapting our method to other tasks, we can finetune the vision encoder to increase the performance.

Reviewer comment (Presentation):
There are some formatting errors, i.e Figure 4.

Author response:
We have revised Figure 4 and update the manuscript.

Reviewer comment (Reproducibility):
Please elaborate on how to match and align between video and text.

Author response:
We use contrastive loss and video text matching loss to supervise the alignment between video and text in stage 1 and stage 2, respectively.  The contrastive loss compare each text to all videos in a batch. And the matching loss compute the text to the hardest video case in a batch with the help of hard negative mining strategy.  Fig. 5 in  supplementary material shows the details of the interaction between text and video in cross attention.

Reviewer comment (Experiments):
In addition to TS2Net, the results of other existing methods are given to better evaluate the performance improvement and advantages of this method.

Author response:
We report the results our method on top of four SOTA cosine similarity methods in Tables 1 and 8 of paper. Specifically on the t2v R@1 metric for the MSRVTT dataset, our method improves by 2.5, 1.2, 3.0, and 1.8 on the Clip4Clip, Xpool, TS2-Net, and clip-vip methods, respectively.

Reviewer comment (Novelty):
For the first stage, this paper uses an existing cosine similarity network, the novelty is limited for me.

Author response:
### 2. For the first stage, this paper uses an existing cosine similarity network, the novelty is limited .

Yes, we clarify Stage 1 is not our contribution. CrossTVR is a  re-ranker which is compatible with existing retrieval methods.

Reviewer comment (Theory):
For the technique part, the motivation for multi-grained video text cross-attention is unclear. How to enhance the fine-grained correspondences by the cross-attention module.

Author response:
### 3. The motivation for multi-grained video text cross-attention is unclear. How to enhance the fine-grained correspondences by the cross-attention module.

In previous work on video text retrieval, one approach is to use the embeddings of video and text for cosine similarity, which is efficient but limited in accuracy. While another is to use all video tokens and text for cross attention, which costs expensive computation. We consider an method that minimizes computational effort while maximizing the retention of frame-level and video-level information to achieve full interaction between video and text in cross attention.  
In the first stage, existing cosine similarity network can retrives Top K most similar results with high recall. Then in the second stage, the goal of re-ranker is to find the minor differents between these K results, which we believe fine grained correspondences are critical. Therefore, we use frame text attention to achieve the interaction between frame and text to obtain fine-grained features. And we further visualize the fine-grained feature attention map to show our fine-grained feature extraction capability in supplymentary meterial Fig. 7.

Reviewer comment (Writing):
Furthermore, what specific attention mechanism is being employed? Please explain this work's fundamental research insight.

Author response:
### 4. What specific attention mechanism is being employed? Please explain this work's fundamental research insight.

Our attention mechanism is built on top of BERT, which is also widely used in previous multimodal work such as ALBEF, BLIP, X-VLM. We show the details of our cross attention in the supplementary material Fig. 5. The attention mechanism is not novel, the way to fuse frame level information and video level information is critical in our method. Specifically, We use a set of  fixed number learnable queries to sequentially  interact with text, frame level , and video level information, which extracts relavent information from each other.

Reviewer comment (Novelty):
The main contributions of this paper are not clear enough to show the advantages and the novelty is not well explained. Please summary the main contributions in the introduction section.

Author response:
### 5. The main contributions of this paper are not clear enough to show the advantages and the novelty is not well explained. Please summary the main contributions in the introduction section.

In summary, The main contributions of this work are as follows:

* We propose a multi-grained re-ranker called CrossTVR, which achieves comprehensive interaction between text and video at the frame level and video level.
* As a re-ranker, our method can be widely applied to existing cosine similarity-based methods and effectively improve the SOTA retrieval performance with marginal additional computation cost.
* Benefiting from our freezing visual coder training method, our approach can scale to larger pretrain visual models with small computational resources.