Paper: Valley: Video Assistant with Large Language model Enhanced abilitY (ICLR-2024)

Reviewer comment (Experiments):
The authors evaluate their model on general video question answering tasks, but on domain-specific image understanding benchmarks, which seems illogical to me. As the authors claim that Vally can comprehend video, image, and language within a general framework, common image question answering datasets such as VQAv2 and GQA should be evaluated.

Author response:
**A**: We acknowledge that our focus is primarily on video understanding, which is why we have fewer experiments specifically related to images. However, we want to emphasize that Valley, in terms of its structure, is capable of comprehending both images and videos. The image case analyses presented in the paper demonstrate this capability. Also, we have evaluated our valley model on LLaVA benchmark，see details in global response. Results have demonstrated its effectiveness on image understanding.

We appreciate your suggestion to include benchmarks such as VQAv2 and GQA in our future work.

Reviewer comment (Experiments):
In terms of instruction dataset construction, there seems to be a lack of innovation and comparative experiments. It appears that a higher-quality data source was simply used to collect data, and then common methods were employed to construct the instruction dataset. This was combined with instruction datasets from previous methods to obtain a larger instruction dataset. The decent performance achieved by this method in quantitative analysis may reflect the quality of the instruction dataset to some extent. However, there has been no comparison of the quality of the instruction dataset collected in this paper with that of previous datasets under fair conditions (such as equal data volume).

Author response:
**A**: We re-trained on the valley-v1 structure using only videochat data (videochat-instruct-11k), and the experimental result on three datasets (MSVD, MSRVTT, and ActivityNet) are detailed in global response. We can see that under the same amount of instruction data (11k), Valley performs better than the videochat model of the same size (13b). It shows that we use simple linear layers to align video and text modalities is effective. And we prove that splice VIT's "[CLS]" token for temporal representation performs well.

And through experiments, we also prove the advantages and disadvantages of different timing modeling methods (valley-v1, valley-v2, valley-v3) under the same amount data, which has guiding significance for future research.

Reviewer comment (Novelty):
In terms of the model methodology, there is a lack of innovation and comparative experiments. This paper attempts three temporal modeling strategies for video input, but these are very basic methods that have been widely used in tasks such as action recognition. There are no effective improvements aimed at enhancing conversational capabilities. Furthermore, the comparison with other methods like VideoChat is not detailed enough. More comparisons are needed (e.g., performance comparison under the same dataset or same backbone) to demonstrate that the temporal modeling and other modules used in this paper have superior performance and efficiency.

Author response:
**A**: We re-trained on the valley-v1 structure using only videochat data (videochat-instruct-11k), and the experimental result on three datasets (MSVD, MSRVTT, and ActivityNet) are detailed in global response. We can see that under the same amount of instruction data (11k), Valley performs better than the videochat model of the same size (13b). It shows that we use simple linear layers to align video and text modalities is effective. And we prove that splice VIT's "[CLS]" token for temporal representation performs well.

And through experiments, we also prove the advantages and disadvantages of different timing modeling methods (valley-v1, valley-v2, valley-v3) under the same amount data, which has guiding significance for future research.

**A**: 
1. The prevalent works, such as VideoChat, Video LLaMA, adopt Q-former to bridge the textual and visual modality, while we turn to the simpler linear projection layer.
2. Similar concurrent works such as Video-ChatGPT also use the linear projection layer, but its use mean pooling on both spatial and temporal tokens for video modeling, which is sub-optimal in building video assistant. We design three temporal modeling methods, and experimentally verified the superiority of valley-v3 in temporal understanding compared to other baselines.
3. We collect and design a large video instruction-following dataset with the assistance of ChatGPT. Experiments demonstrate the quality of our proposed dataset. We have released this dataset 'Valley-702K', and there are already works beneficial from our proposed dataset such as *Video-LLaVA*[1]

Reviewer comment (Experiments):
More fair experimental comparisons are needed to demonstrate that the instruction dataset collected in this paper is of higher quality than previous instruction datasets.

Author response:
**A**: We re-trained on the valley-v1 structure using only videochat data (videochat-instruct-11k), and the experimental result on three datasets (MSVD, MSRVTT, and ActivityNet) are detailed in global response. We can see that under the same amount of instruction data (11k), Valley performs better than the videochat model of the same size (13b). It shows that we use simple linear layers to align video and text modalities is effective. And we prove that splice VIT's "[CLS]" token for temporal representation performs well.

And through experiments, we also prove the advantages and disadvantages of different timing modeling methods (valley-v1, valley-v2, valley-v3) under the same amount data, which has guiding significance for future research.

Reviewer comment (Experiments):
More fair experimental comparisons are needed to demonstrate that the methodology proposed in this paper performs better in terms of performance or efficiency compared to previous methods.

Author response:
**A**: We re-trained on the valley-v1 structure using only videochat data (videochat-instruct-11k), and the experimental result on three datasets (MSVD, MSRVTT, and ActivityNet) are detailed in global response. We can see that under the same amount of instruction data (11k), Valley performs better than the videochat model of the same size (13b). It shows that we use simple linear layers to align video and text modalities is effective. And we prove that splice VIT's "[CLS]" token for temporal representation performs well.

And through experiments, we also prove the advantages and disadvantages of different timing modeling methods (valley-v1, valley-v2, valley-v3) under the same amount data, which has guiding significance for future research.

Reviewer comment (Novelty):
Are there any technical contributions that I have overlooked? In my view, the biggest difference between Valley and previous methods is the instruction dataset used.

Author response:
**A**: 
1. The prevalent works, such as VideoChat, Video LLaMA, adopt Q-former to bridge the textual and visual modality, while we turn to the simpler linear projection layer.
2. Similar concurrent works such as Video-ChatGPT also use the linear projection layer, but its use mean pooling on both spatial and temporal tokens for video modeling, which is sub-optimal in building video assistant. We design three temporal modeling methods, and experimentally verified the superiority of valley-v3 in temporal understanding compared to other baselines.
3. We collect and design a large video instruction-following dataset with the assistance of ChatGPT. Experiments demonstrate the quality of our proposed dataset. We have released this dataset 'Valley-702K', and there are already works beneficial from our proposed dataset such as *Video-LLaVA*[1]

Reviewer comment (Experiments):
While the data collection pipeline is well-formulated, this method requires very high-quality training data. Gathering high-quality video instruction data remains very challenging when aiming for large-scale training. This prohibits very large-scale training to significantly boosting the model quality, especially when it comes to the video domain where the video data is often sparse and requires a very large number of training data.

Author response:
**A**: We completely agree with your viewpoint and acknowledge the importance of high-quality video instruction data for our method. We are also well aware that video data in the domain is often sparse and requires a substantial amount of high-quality training data.  Therefore, in our current research, we have made significant efforts to gather the available high-quality data from open websites and design instruction and conversations to train the model. Based on our knowledge, it is of the largest scale in existing video instruction dataset. And the results prove its "high quality". We hope the data will be helpful to the community and call for researchers to seek additional resources and collaborations to obtain more high-quality video data from the real world.

Reviewer comment (Experiments):
The direct integration of vision transformers and LLMs may encounter obstacles, especially when dealing with lengthy videos. While the method was tested on some benchmarks, the videos tested in this paper are often very short (only a few seconds). There is still a long distance to real-world video applications.

Author response:
**A**: Thank you for your insightful comments regarding the misalignment between the Vision Transformer (ViT) and the Language and Vision Model (LLM). To address this issue, ViT from CLIP which has undergone text-to-vision alignment training is selected and we have pretrained a projection layer with large amount of video-text pairs and image-text pairs to effectively align the representations. Regarding the evaluation datasets, we selected them based on wide usage in video understanding research. However, we acknowledge the importance of considering a diverse range of evaluation datasets, especially lengthy videos, and will take your suggestion into account for future research.  

For the challenge of video length variation, we designed valley-v3. The valley-v3 we designed supports video input of varying lengths (video sampling based on fps). In practical applications, for longer videos, the sampling fps value can be reduced.

And for your second concern, your understanding of this issue is very accurate. Indeed, evaluating the capabilities of multimodal large models is a challenging problem. Existing benchmark tests cannot fully cover all scenarios and cannot exhaustively capture all aspects of perceptual analysis. Therefore, it is a relatively reasonable approach to manually collect video data from the real world and construct evaluation sets from multiple dimensions to assess the understanding and reasoning abilities of large models. However, this method may introduce subjectivity and may not be widely accepted by everyone, and it requires a significant amount of human resources. Currently, it is difficult for us to achieve this, but we can work together to continuously improve and refine evaluation methods.

Reviewer comment (Novelty):
It is not clear what is the technical novelty of the proposed method Valley. Throughout the introduction, related works, and method sections there is not statement that explains the technical difference distinct from the existing video-language models.

Author response:
**A**: We propose employing a simple linear projection layer as a bridge between the video and text modalities, while similar works mostly adopt the complex Q-former structure. Besides, we specifically design three temporal modeling modules  for improving video understanding.

Reviewer comment (Evaluation):
Among the temporal modeling modules, what is the unique advantage of the version-2 with linear layers over v1 and v3? There seems no benchmark where the v2 performs the best and so it is not clear why v2 is proposed as one of the main methods.

Author response:
**A**: For improving v1, v2 introduces  a learnable linear layer to learn the temporal importance score of a certain frame. In order to more fully model the temporal variation representation of these spatial tokens, we derived the v3 based on Transformer. In light of experiments, the performance of V2 is basically between V1 and V3, such as the MSRVTT-QA and the video-based text generation benchmark provided by Video-ChatGPT. Refer to Table 1 and Table 2.

Reviewer comment (Reproducibility):
There is no implementation details. It seems not easy to reproduce the result based on what is provided in the main manuscript.

Author response:
**A**: We will release all the source code to facilitate the reproduction of the results in the paper.

Reviewer comment (Experiments):
What is the model size and runtime comparison with other state-of-the-art methods (VideoChat, Video LLaMA, Video-ChatGPT)?

Author response:
**A**: Model size of VideoChatGPT and LLaMA-adaptor is 7B, and others are all 13B. We did not perform the inference of all these models and the evaluation metrics are claimed from the papers.