|
--- |
|
title: COVER |
|
emoji: 🏃 |
|
colorFrom: blue |
|
colorTo: yellow |
|
sdk: gradio |
|
sdk_version: 4.36.1 |
|
python_version: 3.9 |
|
app_file: app.py |
|
pinned: false |
|
license: mit |
|
--- |
|
|
|
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference |
|
|
|
# 🏆 [CVPRW 2024] [COVER](https://openaccess.thecvf.com/content/CVPR2024W/AI4Streaming/papers/He_COVER_A_Comprehensive_Video_Quality_Evaluator_CVPRW_2024_paper.pdf): A Comprehensive Video Quality Evaluator. |
|
|
|
🏆 🥇 **Winner solution for [Video Quality Assessment Challenge](https://codalab.lisn.upsaclay.fr/competitions/17340) at the 1st [AIS 2024](https://ai4streaming-workshop.github.io/) workshop @ CVPR 2024** |
|
|
|
Official Code for [CVPR Workshop 2024] Paper *"COVER: A Comprehensive Video Quality Evaluator"*. |
|
Official Code, Demo, Weights for the [Comprehensive Video Quality Evaluator (COVER)](https://openaccess.thecvf.com/content/CVPR2024W/AI4Streaming/papers/He_COVER_A_Comprehensive_Video_Quality_Evaluator_CVPRW_2024_paper.pdf). |
|
|
|
- 29 May, 2024: We create a space for [COVER](https://huggingface.co/spaces/Sorakado/COVER) on Hugging Face. |
|
- 09 May, 2024: We upload Code of [COVER](https://github.com/vztu/COVER). |
|
- 12 Apr, 2024: COVER has been accepted by CVPR Workshop2024. |
|
|
|
 [](https://github.com/vztu/COVER) |
|
[](https://github.com/vztu/COVER) |
|
<a href="https://huggingface.co/spaces/Sorakado/COVER"><img src="./figs/deploy-on-spaces-sm-dark.svg" alt="hugging face log"></a> |
|
|
|
## Introduction |
|
- Existing UGC VQA models strive to quantify quality degradation mainly from technical aspect, with a few considering aesthetic or semantic aspects, but no model has addressed all three aspects simultaneously. |
|
- The demand for high-resolution and high-frame-rate videos on social media platforms presents new challenges for VQA tasks, as they must ensure effectiveness while also meeting real-time requirements. |
|
|
|
## the proposed COVER |
|
|
|
*This inspires us to develop comprehensive and efficient model for UGC VQA task* |
|
|
|
 |
|
|
|
### COVER |
|
|
|
Results comparison: |
|
| Dataset: YT-UGC | SROCC | KROCC | PLCC | RMSE | Run Time | |
|
| ---- | ---- | ---- | ---- | ---- | ---- | |
|
| [**COVER**](https://github.com/vztu/COVER/release/Model/COVER.pth) | 0.9143 | 0.7413 | 0.9122 | 0.2519 | 79.37ms | |
|
| TVQE (Wang *et al*, CVPRWS 2024) | 0.9150 | 0.7410 | 0.9182 | ------- | 705.30ms | |
|
| Q-Align (Zhang *et al, CVPRWS 2024) | 0.9080 | 0.7340 | 0.9120 | ------- | 1707.06ms | |
|
| SimpleVQA+ (Sun *et al, CVPRWS 2024) | 0.9060 | 0.7280 | 0.9110 | ------- | 245.51ms | |
|
|
|
The run time is measured on an NVIDIA A100 GPU. A clip |
|
of 30 frames of 4K resolution 3840×2160 is used as input. |
|
|
|
## Install |
|
|
|
The repository can be installed via the following commands: |
|
```shell |
|
git clone https://github.com/vztu/COVER |
|
cd COVER |
|
pip install -e . |
|
mkdir pretrained_weights |
|
cd pretrained_weights |
|
wget https://github.com/vztu/COVER/release/Model/COVER.pth |
|
cd .. |
|
``` |
|
|
|
|
|
## Evaluation: Judge the Quality of Any Video |
|
|
|
### Try on Demos |
|
You can run a single command to judge the quality of the demo videos in comparison with videos in VQA datasets. |
|
|
|
```shell |
|
python evaluate_one_video.py -v ./demo/video_1.mp4 |
|
``` |
|
|
|
or |
|
|
|
```shell |
|
python evaluate_one_video.py -v ./demo/video_2.mp4 |
|
``` |
|
|
|
Or choose any video you like to predict its quality: |
|
|
|
|
|
```shell |
|
python evaluate_one_video.py -v $YOUR_SPECIFIED_VIDEO_PATH$ |
|
``` |
|
|
|
### Outputs |
|
|
|
The script can directly score the video's overall quality (considering all perspectives). |
|
|
|
```shell |
|
python evaluate_one_video.py -v $YOUR_SPECIFIED_VIDEO_PATH$ |
|
``` |
|
|
|
The final output score is the sum of all perspectives. |
|
|
|
|
|
## Evaluate on a Exsiting Video Dataset |
|
|
|
|
|
```shell |
|
python evaluate_one_dataset.py -in $YOUR_SPECIFIED_DIR$ -out $OUTPUT_CSV_PATH$ |
|
``` |
|
|
|
## Evaluate on a Set of Unlabelled Videos |
|
|
|
|
|
```shell |
|
python evaluate_a_set_of_videos.py -in $YOUR_SPECIFIED_DIR$ -out $OUTPUT_CSV_PATH$ |
|
``` |
|
|
|
The results are stored as `.csv` files in cover_predictions in your `OUTPUT_CSV_PATH`. |
|
|
|
Please feel free to use COVER to pseudo-label your non-quality video datasets. |
|
|
|
|
|
## Data Preparation |
|
|
|
We have already converted the labels for most popular datasets you will need for Blind Video Quality Assessment, |
|
and the download links for the **videos** are as follows: |
|
|
|
:book: LSVQ: [Github](https://github.com/baidut/PatchVQ) |
|
|
|
:book: KoNViD-1k: [Official Site](http://database.mmsp-kn.de/konvid-1k-database.html) |
|
|
|
:book: LIVE-VQC: [Official Site](http://live.ece.utexas.edu/research/LIVEVQC) |
|
|
|
:book: YouTube-UGC: [Official Site](https://media.withyoutube.com) |
|
|
|
*(Please contact the original authors if the download links were unavailable.)* |
|
|
|
After downloading, kindly put them under the `../datasets` or anywhere but remember to change the `data_prefix` respectively in the [config file](cover.yml). |
|
|
|
# Training: Adapt COVER to your video quality dataset! |
|
|
|
Now you can employ ***head-only/end-to-end transfer*** of COVER to get dataset-specific VQA prediction heads. |
|
|
|
```shell |
|
python transfer_learning.py -t $YOUR_SPECIFIED_DATASET_NAME$ |
|
``` |
|
|
|
For existing public datasets, type the following commands for respective ones: |
|
|
|
- `python transfer_learning.py -t val-kv1k` for KoNViD-1k. |
|
- `python transfer_learning.py -t val-ytugc` for YouTube-UGC. |
|
- `python transfer_learning.py -t val-cvd2014` for CVD2014. |
|
- `python transfer_learning.py -t val-livevqc` for LIVE-VQC. |
|
|
|
As the backbone will not be updated here, the checkpoint saving process will only save the regression heads. To use it, simply replace the head weights with the official weights [COVER.pth](https://github.com/vztu/COVER/release/Model/COVER.pth). |
|
|
|
We also support ***end-to-end*** fine-tune right now (by modifying the `num_epochs: 0` to `num_epochs: 15` in `./cover.yml`). It will require more memory cost and more storage cost for the weights (with full parameters) saved, but will result in optimal accuracy. |
|
|
|
## Visualization |
|
|
|
### WandB Training and Evaluation Curves |
|
|
|
You can be monitoring your results on WandB! |
|
|
|
## Acknowledgement |
|
|
|
Thanks for every participant of the subjective studies! |
|
|
|
## Citation |
|
|
|
Should you find our work interesting and would like to cite it, please feel free to add these in your references! |
|
|
|
```bibtex |
|
%AIS 2024 VQA challenge |
|
@article{conde2024ais, |
|
title={AIS 2024 challenge on video quality assessment of user-generated content: Methods and results}, |
|
author={Conde, Marcos V and Zadtootaghaj, Saman and Barman, Nabajeet and Timofte, Radu and He, Chenlong and Zheng, Qi and Zhu, Ruoxi and Tu, Zhengzhong and Wang, Haiqiang and Chen, Xiangguang and others}, |
|
journal={arXiv preprint arXiv:2404.16205}, |
|
year={2024} |
|
} |
|
|
|
%cover |
|
@article{cover2024cpvrws, |
|
title={COVER: A comprehensive video quality evaluator}, |
|
author={Chenlong, He and Qi, Zheng and Ruoxi, Zhu and Xiaoyang, Zeng and |
|
Yibo, Fan and Zhengzhong, Tu}, |
|
journal={In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops}, |
|
year={2024} |
|
} |
|
``` |
|
|
