Paper: Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks (ICLR-2024)

Reviewer comment (Experiments):
After downloading the dataset, it was found that there were many duplicate clips from the same source and static clips. Does the situation exist where these 400 million video clips come from the same original video? If so, during the filtering process, how is the quality of the selected videos ensured given the lack of quantifiable performance measures, such as CLIP similarity?

Author response:
## Duplicate and Source

**Duplicate**: Thank you for the suggestion. To count the number of static videos, we sample frames from each video and then compute the difference between frames from the same video. Our statistical analysis revealed that a significant majority, 99.6%, of the videos were dynamic, with only a small proportion being entirely static with dynamic audio.

**Source**: The 400 million videos mentioned correspond to the entire content pool of short videos on the Youku website. This vast collection primarily consists of user-generated uploads and platform-generated content. In Section 3.1, we have elucidated the process involved in constructing the pre-training dataset. The videos are carefully classified into 45 diverse categories covering various domains, e.g., Daily life, Comedy, and Pet, with a balanced distribution. If there are any aspects of this that remain unclear, we would be more than happy to provide further clarification.

Reviewer comment (Experiments):
There is a lack of exploration into the status of text annotation in the dataset. Chinese and Latin languages such as English have significant differences in vocabulary, grammar, and sentence structure. The diversity of the text part of this dataset is not sufficiently demonstrated, and the text quality is slightly lower compared to the WebVid10M dataset. The paper should also compare the dataset with other existing video-language datasets, such as translated HowTo100M, WebVid10M or CNVid-3.5M[1], and discuss the advantages and limitations of the dataset.

Author response:
## Text Annotation

### Translation

There are several problems with translating English video data into Chinese.

1. Due to the limitations of translation tools and the proficiency of annotators, the translated text often does not take into account the usage habits of Chinese language, and may even introduce errors. To verify this, we used GPT-3.5 to automatically translate a few of captions from webvid2m, and demonstrated the problems caused by translation. For example, the caption "Woman showing victory sign, indoor" was translated as "室内女性展示胜利手势". The translation result does not align with the actual Chinese language conventions in terms of word choice and language habits. A more natural Chinese expression should be "房间里，一个女性正在比耶". Another example is the translation of "Cottage village surrounded by mountains on the shores of the Atlantic Ocean" as "小屋村庄环绕在大西洋海岸的山脉上", where the translated text expresses mountains surrounded by a village, which is completely different from the original meaning in the English text. These inaccuracies greatly impact the training of the model.

2. We further validated the cultural and linguistic differences between the two by analyzing the word clouds of keywords (translated to Chinese) in the webvid2m dataset (Figure 1) and video keywords in mPLUG-Youku (Figure 2). The webvid2m dataset contains a large number of keywords related to places (e.g., Moscow, Bangkok, Spain) and time periods (e.g., 2019, 2015, 1950s). In Chinese videos, the high-frequency words are mainly related to variety shows (e.g., "王牌对王牌" which translates to "Ace VS Ace"), games (e.g., "英雄联盟" which translates to "League of Legends"), film and television works (e.g., "乡村爱情" which translates to "Country Love"), and celebrities (e.g., "张大仙" which translates to "Zhang Daxian", "岳云鹏" which translates to "Yue Yunpeng"). It can be seen that Chinese videos cover a large number of named entities related to Chinese culture, which is significantly different from the distribution in English videos. If only translated videos are used for training, the model will lack the ability to understand these types of named entities.

**Figure 1**
![Figure 1](https://z1.ax1x.com/2023/11/16/pitJMPH.png)

**Figure 2**
![Figure 2](https://z1.ax1x.com/2023/11/16/pitJF2R.png)

3. Although English text can be translated into Chinese, it is difficult to translate audio (e.g., narrations, dialogues) and text in the form of images in the video into Chinese. The learned textual-visual relationship from English content may lead to domain inconsistency when applied to Chinese content.

### Diversity and Quality

For the pre-training dataset, we filter 10 million high-quality video-text pairs from 400 million raw videos with strict safety, diversity, and quality criteria. In chapter 3.1, we have provided a detailed explanation of the process of constructing pre-training dataset.  Especially, we employ the Chinese image-text pre-trained model CLIP to improve the data quality by deprecating those with low similarities between the mean frame features and text features. The videos are carefully classified into 45 diverse categories covering various domains, e.g., Daily life, Comedy, and Pet, with a balanced distribution in Figure 2,3. Figure3 shows the distribution of clip score.

**Figure 3**
![Figure 3](https://z1.ax1x.com/2023/11/16/pitJNdS.png)

Reviewer comment (Experiments):
This paper only explores the zero-shot capability in instruction understanding. Why not further investigate the zero-shot performance in video classification, retrieval, and description?

Author response:
## Zeroshot

we directly tested the results of zero-shot methods with pretraining. 

**Zero-shot Caption w pretraining**

||BLEU4|METEOR|ROUGE_L|CIDEr|
|-|-|-|-|-|
|mPLUG-2|5.9|6.2|13.2|49.2|
|mPLUG-video (1.3B)|7.1|8.5|17.37|58.9|

**Zero-shot retrieval w pretraining**

| |T2V R1|T2V R5|T2V R10|V2T R1|V2T R5|V2T R10
|-|-|-|-|-|-|-|
ALPRO|14.17|31.48|39.69|10.43|24.68|31.98
mPLUG-2|18.14|35.82|43.93|18.14|35.82|43.93

Reviewer comment (Evaluation):
In instruction understanding, does VideoLLaMA also receive Chinese prompts? Has it been trained on Chinese instruction data? Comparing a MLLM trained on English datasets with one training in Chinese is unfair.

Author response:
## Evaluation of VideoLLaMA

For VideoLLaMA, we evaluate the performance of model trained with bilingual backbone (i.e., BiLLA) in our comparison. 

Besides, VideoLLaMA utilizes the video instruction data during instructional tuning, while our mPLUG-video only leverages fewer image instruction data than those of VideoLLaMA. Moreover, due to the lack of Chinese instruction data, we utilize the English instruction only, which further demonstrates the effectiveness of our proposed method.

Reviewer comment (Evaluation):
During data collection, the online model achieved a performance of about 94% in video category classification. However, in Table 4, the model trained by Youku-mPLUG actually performs worse than the unfiltered online model.

Author response:
## Performance of Online Model

The online model we utilize is not a standalone entity, but rather a dedicated pipeline composed of several models and diverse input signals. Specifically, the online model integrates multiple classification models that use user behavior data, similarity retrieval, and user annotations as multi-dimensional information. In contrast, our mPLUG-video operates independently as a single model trained on the downstream task dataset, devoid of any additional features or complexities.

If you have any more questions or concerns, please feel free to further discuss.

Reviewer comment (Evaluation):
The improvements brought by the proposed mPLUG-video are limited.

Author response:
Thank you for recognizing the quality of our dataset and the technology we have employed. Below, we will first elaborate on the novelty and improvements of our contribution and thoroughly explain and validate the quality of our dataset. Finally, we will use the webvid dataset as an example to demonstrate that relying solely on translated data is far from sufficient.

## Novelty and improvement

**Open-sourcing**: With the overarching aim of advancing the development of Chinese multimodal technology, our work primarily revolves around open-sourcing the largest Chinese video pre-training dataset and establishing a robust benchmark complemented by human-written ground truth. The data set has already been released on the open-source platform and been downloaded by more than 26k times. Due to the anonymity policy, we are not sure if we can provide the link during rebuttal, but will definitely attach the link if the work can be accepted. We are committed to refining the download process continually and have introduced features like streaming and batch downloads to enhance user convenience.

**Models**: we offer comprehensive benchmark evaluations of models across a variety of architectures. This includes encoder-only models (ALPRO), encoder-decoder models (mPLUG-2), and decoder-only models (mPLUG-Video) for comparison purposes.

**MLLM**: we've trained the first Chinese Multimodal LLM specifically for video understanding. To validate the effectiveness of pretraining on our datasets, we have conducted an extensive range of experiments. These include both benchmark evaluations and human assessments.

**Performance**: While the development of a novel model isn't the primary focus of our work, our experiments do yield valuable insights into creating more effective models, such as the potential benefits of increasing model size (as presented in Tables 4 and 6) as well as exploring modality complementarity (as shown in Table 6).

Reviewer comment (Experiments):
One of the key contributions in this paper is the proposed new dataset. It would be better to demonstrate the high quality of the newly collected data. Based on the example shown in Figure 9, the text annotations look very noisy.

Author response:
## Data Quality

**Contruction**: For the pre-training dataset, we filter 10 million high-quality video-text pairs from 400 million raw videos with strict safety, diversity, and quality criteria. In Section 3.1, we have provided a detailed explanation of the process of constructing pre-training dataset.  Especially, we employ the Chinese image-text pre-trained model CLIP to improve the data quality by deprecating those with low similarities between the mean frame features and text features. Figure 1 shows the distribution of clip score.

**Figure1**
![Figure1](https://z1.ax1x.com/2023/11/16/pitJNdS.png)

**Common Practice**: The pre-training dataset is primarily aimed at improving the diversity and generalization of the model. Noise is inevitable when generating a huge data set and can be observed in many prevalent data sets, e.g., ImageNet, LAION, etc. While noise may influence the learning process, empirical results show that pre-training is robust to a small amout of noise and still can obtain impressive models from them, e.g., OpenCLIP by training on LAION. Our experiments also confirms the effectiveness of the proposed data set in Table 5.

**Performance**: Table 5 also demonstrates models can benefit from pre-training on Youku-mPLUG. In specific, by pretraining on Youku-mPLUG, the model's performance has increased by 20.4 point of CIDEr on VATEX datasets. Meanwhile, the result in Table 7 also demonstrates the effectiveness of our proposed dataset as the accuracy has increased 8.7 points on the classification task.

**Downstream Task**: The data quality of downstream tasks is  high. For each downstream task, we hire well-educated people and adopt a two-step verification to ensure the quality and diversity of the annotations.

**Open-sourcing**: Our work mainly focuses on open-sourcing the largest Chinese video pre-training dataset and a well-developed benchmark with human-written ground truth. The data set has already been released on the open-source platform and been downloaded by more than 26k times. Due to the anonymity policy, we are not sure if we can provide the link during rebuttal, but will definitely attach the link if the work can be accepted . We have been continuously optimizing the download process, including adding streaming and batching downloads.

Reviewer comment (Writing):
In the first paragraph of the introduction section, the authors argue that existing methods of translating English to Chinese suffer intrinsic linguistic and cultural gaps. Could you give more explicit examples to show the harmfulness of these methods?

Author response:
## Translation

There are several problems with translating English video data into Chinese.

1. Due to the limitations of translation tools and the proficiency of annotators, the translated text often does not take into account the usage habits of Chinese language, and may even introduce errors. To verify this, we used GPT-3.5 to automatically translate a few of captions from webvid2m, and demonstrated the problems caused by translation. For example, the caption "Woman showing victory sign, indoor" was translated as "室内女性展示胜利手势". The translation result does not align with the actual Chinese language conventions in terms of word choice and language habits. A more natural Chinese expression should be "房间里，一个女性正在比耶". Another example is the translation of "Cottage village surrounded by mountains on the shores of the Atlantic Ocean" as "小屋村庄环绕在大西洋海岸的山脉上", where the translated text expresses mountains surrounded by a village, which is completely different from the original meaning in the English text. These inaccuracies greatly impact the training of the model.
2. We further validated the cultural and linguistic differences between the two by analyzing the word clouds of keywords (translated to Chinese) in the webvid2m dataset (Figure 2) and video keywords in mPLUG-Youku (Figure 3). The webvid2m dataset contains a large number of keywords related to places (e.g., Moscow, Bangkok, Spain) and time periods (e.g., 2019, 2015, 1950s). In Chinese videos, the high-frequency words are mainly related to variety shows (e.g., "王牌对王牌" which translates to "Ace VS Ace"), games (e.g., "英雄联盟" which translates to "League of Legends"), film and television works (e.g., "乡村爱情" which translates to "Country Love"), and celebrities (e.g., "张大仙" which translates to "Zhang Daxian", "岳云鹏" which translates to "Yue Yunpeng"). It can be seen that Chinese videos cover a large number of named entities related to Chinese culture, which is significantly different from the distribution in English videos. If only translated videos are used for training, the model will lack the ability to understand these types of named entities.

**Figure 2**
![Figure 2](https://z1.ax1x.com/2023/11/16/pitJMPH.png)

**Figure 3**
![Figure 3](https://z1.ax1x.com/2023/11/16/pitJF2R.png)

3. Although English text can be translated into Chinese, it is difficult to translate audio (e.g., narrations, dialogues) and text in the form of images in the video into Chinese. The learned textual-visual relationship from English content may lead to domain inconsistency when applied to Chinese content.

Reviewer comment (Experiments):
The experiments are not very comprehensive. The selected baseline models in different downstream tasks is limited, two were selected only.

Author response:
Due to the limited time for rebuttal and the high cost of pre-training, we directly tested the results of fine-tuning baselines. 

| |T2V R1|T2V R5|T2V R10|V2T R1|V2T R5|V2T R10
|-|-|-|-|-|-|-|
HiTeA|9.42|27.47|38.38|9.99|28.04|38.54
ALPRO|8.90    |26.76|36.29|8.87|26.45|37.80
mPLUG-2|11.46|28.24|39.06|10.21|30.27|41.75

In addition, Table5 also demonstrates models can benefit from pre-training on Youku-mPLUG with 20.4 improvement. It would be greatly appreciated if you could provide the details about the appropriate baselines, e.g., name of papers. We will included them in the revised version.

Reviewer comment (Experiments):
No details about the selection of the original 400 million videos are provided.

Author response:
The 400 million videos mentioned correspond to the entire content pool of short videos on the Youku website. This vast collection primarily consists of user-generated uploads and platform-generated content. In Section 3.1, we have elucidated the process involved in constructing the pre-training dataset. If there are any aspects of this that remain unclear, we would be more than happy to provide further clarification.

Reviewer comment (Reproducibility):
This paper mentions that currently existing large-scale Chineses video-language datasets are not publicly accessible. This also demonstrates that not only the collection and curation of large datasets are challenging, but the release process is also difficult. Could the authors provide their plans to prove that you can genuinely release this dataset and make it easily accessible to researchers, thus making a real contribution to the research community?

Author response:
We appreciate your acknowledgement of our efforts in constructing the dataset and our contributions to the open-source community. The data set has already been released on the open-source platform and been downloaded by more than 26k times. Due to the anonymity policy, we are not sure if we can provide the link during rebuttal, but will definitely attach the link if the work can be accepted.  We are committed to continuously optimizing the download process to make it more user-friendly, which includes the implementation of features such as streaming and batch downloads.

Reviewer comment (Experiments):
The results in Table 5 are not convincing enough. The authors only compare one publicly available dataset VATEX and do not show a gap with current state-of-the-art results.

Author response:
Due to the lack of open-sourcing Chinese pre-training datasets and benchmarks, there are very few public Chinese multimodal models. There is no existing method to evaluate the Vatex Chinese Caption dataset. Our mPLUG-Video model based on the Youku dataset is currently the state-of-the-art (SOTA) method for the Vatex Chinese Caption dataset. 

And we reproduced classic English models with different structures (encoder-only (i.e., ALPRO), encoder-decoder (i.e., mPLUG-2), and decoder-only (i.e., mPLUG-Video)) for comparison. Due to limited computing power and high pre-training costs, we have only reproduced these baselines.  Any suggestions about other baselines are welcome and can be included in the revised version.

Reviewer comment (Experiments):
Data augmentation will almost certainly bring performance improvements to the model. Therefore, how to prove that Youku-mPLUG is superior to other dataset like CNVid-3.5M?

Author response:
1. Open-sourcing: Our work mainly focuses on open-sourcing the largest Chinese video pre-training dataset and a well-developed benchmark with human-written ground truth. The data set has already been released on the open-source platform and been downloaded by more than 26k times. Due to the anonymity policy, we are not sure if we can provide the link during rebuttal, but will definitely attach the link if the work can be accepted. We have been continuously optimizing the download process, including adding streaming and batching downloads. There is no open-source model and code for CNVid-3.5M, and the pre-training dataset has not been open-sourced (https://github.com/CNVid/CNVid-3.5M/issues)
2. Benchmark: we carefully build the largest human-annotated Chinese benchmarks covering three popular video-language tasks across cross-modal retrieval, video captioning, and video category classification.
3. Performance: There is no open-source model and code for CNVid-3.5M, and the pre-training dataset has not been open-sourced (https://github.com/CNVid/CNVid-3.5M/issues). Therefore, we evaluate the results of a structure similar to mPLUG-2 on VATEX and compare it with the model based on CNVid-3.5M. We can find that the models based on our dataset achieve better performance on VATEX (R1+2.7, R5+3.8, R10+2.3).

| |R1|R5|R10
|-|-|-|-|
CNVid-3.5M (w/o CNVid-3.5M pt)|36.4|75.4|85.2
mPLUG-2 (w/o Youku pt)|36.3|74.8|85.7
CNVid-3.5M (w CNVid-3.5M pt)|41.5|78.2|87.2
mPLUG-2 (w Youku pt)|44.2|82.0|89.5

4. Models: We also provide comprehensive benchmark evaluations of models across different architectures including encoder-only (i.e., ALPRO), encoder-decoder (i.e., mPLUG-2), and decoder-only (i.e., mPLUG-Video) for comparison. Especially, we train the first Chinese Multimodal LLM for video understanding and conduct comprehensive experiments (both benchmark and human evaluation) to validate the effectiveness of pretraining on our datasets.

If you have any more questions or concerns, please feel free to further discuss.