Repository: Vchitect/VBench
Issue: How to run inference for a new video?
Root cause: Documentation Deficiency

Hello, i'm trying to run inference. I've installed the package. I've tried the code [here](https://github.com/Vchitect/VBench?tab=readme-ov-file#usage).

I have a new directory of videos, called "videos". 
```
(venv) (base) yonatan:~/VBench$ ls videos/
Iron_Man.mp4  birthday.mp4  lavie_human_action_full_info.json  skateboarding_dog.mp4
```
When I try to run 
```
vbench evaluate --videos_path "videos" --dimension "human_action"
```
I receive an error indicating that no data is found. 
When I try through the code, I also get a long list of missing videos:
```python
>>> from vbench import VBench
>>> my_VBench = VBench('cuda', 'VBench_full_info.json', 'videos')
>>> my_VBench.evaluate(videos_path='videos', name='my_test', dimension_list=['human_action'])
WARNING!!! This required video is not found! Missing benchmark videos can lead to unfair evaluation result. The missing video is: A person is riding a bike-0.mp4
...
...
```

I guess that your code only supports your existing evaluated videos? And the instructions do not yet support inference on new videos? 

The question is how can I run the evaluation for a new video + description. Thank you very much 🙏 

Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.

Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json 
A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.

Thank you very much for your great work! looking forward for instructions for evaluating a new video :) 

> Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.
> 
> Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.

If I use my video name replace the prompts in "prompt_en" , but I can't supply the "auxiliary_info", will this have any impact?
Looking forward for you replay.
Thanks!!

> > Thanks for the question and suggestion! We will provide instructions for evaluating a new video soon.
> > Currently, we enforce evaluation on the standard VBench prompt lists to ensure fair comparison among different video generation models. That's also why we give warnings when a required video is not found. This is done via defining the set of prompts in: https://github.com/Vchitect/VBench/blob/master/VBench_full_info.json A quick fix is to define your own VBench_full_info.json file: replacing the prompts in "prompt_en" by your video name.
> 
> If I use my video name replace the prompts in "prompt_en" , but I can't supply the "auxiliary_info", will this have any impact? Looking forward for you replay. Thanks!!

@moclimb  Thank you for your interest in our work! In `['subject_consistency', 'background_consistency', 'temporal_flickering', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', 'imaging_quality', 'temporal_style', 'overall_consistency']`, you don't need to fill in any `auxiliary_info`, you just need to fill in the `dimension` corresponding to your prompt

In `['object_class', 'multiple_objects', 'human_action', 'color', 'spatial_relationship', 'scene', 'appearance_style']`, you need to fill in `auxiliary_info` in accordance with the corresponding rules. You must note that the object class and multiple objects in the tuple must be an object of [COCO](https://cocodataset.org/). The human action must be one action of [Kinetics-400](https://[arxiv.org/pdf/1705.06950v1.pdf](https://arxiv.org/pdf/1705.06950v1.pdf)).

> Hello, i'm trying to run inference. I've installed the package. I've tried the code [here](https://github.com/Vchitect/VBench?tab=readme-ov-file#usage).
> 
> I have a new directory of videos, called "videos".
> 
> ```
> (venv) (base) yonatan:~/VBench$ ls videos/
> Iron_Man.mp4  birthday.mp4  lavie_human_action_full_info.json  skateboarding_dog.mp4
> ```
> 
> When I try to run
> 
> ```
> vbench evaluate --videos_path "videos" --dimension "human_action"
> ```
> 
> I receive an error indicating that no data is found. When I try through the code, I also get a long list of missing videos:
> 
> ```python
> >>> from vbench import VBench
> >>> my_VBench = VBench('cuda', 'VBench_full_info.json', 'videos')
> >>> my_VBench.evaluate(videos_path='videos', name='my_test', dimension_list=['human_action'])
> WARNING!!! This required video is not found! Missing benchmark videos can lead to unfair evaluation result. The missing video is: A person is riding a bike-0.mp4
> ...
> ...
> ```
> 
> I guess that your code only supports your existing evaluated videos? And the instructions do not yet support inference on new videos?
> 
> The question is how can I run the evaluation for a new video + description. Thank you very much 🙏

Hi Yonatan, we now support evaluating user-customised videos: https://github.com/Vchitect/VBench?tab=readme-ov-file#new-evaluate-your-own-videos. You can do this by simply turning the `custom_input ` flag on.  

This new feature now supports those "video quality" dimensions, like temporal consistency, and frame-wise quality. For dimensions that heavily rely on the text prompt, sometimes VBench utilize pre-defined metadata, like the color labels for objects associated with our pre-defined prompt list. Therefore, evaluating videos from user-customised prompts is not yet supported for these prompt-dependent dimensions, but we are working on it. We will gradually enable customised evaluation of more "video-text alignment" dimensions.

I'm closing the issue, feel free to re-open it if you still encountered issues when evaluating new videos.