Repository: Vchitect/VBench
Issue: single dimension running problem
Root cause: Documentation Deficiency

Hi, the path "prompts/prompts_per_dimension" contains only a part of evaluation dimensions. So, if i want to evaluation the dimension of "background_consistent", and there is no proper prompt text file (maybe background_consistent.txt) for me to generation videos. So, what can i do to perform evaluation on "background_consistent"?



And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

> Hi, the path "prompts/prompts_per_dimension" contains only a part of evaluation dimensions. So, if i want to evaluation the dimension of "background_consistent", and there is no proper prompt text file (maybe background_consistent.txt) for me to generation videos. So, what can i do to perform evaluation on "background_consistent"?

@GFENGG  Hi , you can refer to this file. On the left are the corresponding evaluation dimensions, and on the right is the prompt list of left dimension.

For example, if you want to test `background_consistency`, you only need to sample the video of `scene`.

https://github.com/Vchitect/VBench/blob/8046fcbfa1b853d6121956b9f48021de5f1fecb0/dimension_to_folder.json#L2-L17

> And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

@GFENGG  `all_category` is not a standalone dimension, so it will not appear in the `json`. 

`prompts/all_category.txt` is a collection of video prompts for all dimensions that need to be tested. 
By sampling all prompts from `prompts/all_category.txt` into videos, `VBench` will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.


> > And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?
> 
> @GFENGG `all_category` is not a standalone dimension, so it will not appear in the `json`.
> 
> `prompts/all_category.txt` is a collection of video prompts for all dimensions that need to be tested. By sampling all prompts from `prompts/all_category.txt` into videos, `VBench` will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.

Thanks for your reply. In which file (maybe a .json file like "VBench_full_info.json") can i find the relation between the prompts in "prompts/all_category.txt" and corresponding evaluation dimensions?

@GFENGG  Sorry for the late reply. In `all_category`, there are approximately 100 prompts for each category. We conducted VBench evaluation for all prompts in `subject consistency`, `background consistency`, `motion smoothness`, `dynamic degree`, `aestheic quality`, `image quality` and `overall consistency`  without further screening corresponding relationships.

Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.