zhiminy/EvalEng huggingface dataset:
Repository: Vchitect/VBench
Issue: zero output
Root cause: Environment Incompatibility

I want to use this to evaluate my own videos, but it's strange that the output is all 0. Could you help me to correctly use the metricks to eval, thanks!

the code i use:
```bash
python evaluate.py \
 --videos_path ./VBench/11_29_3s \
 --dimension "motion_smoothness" \
 --mode "custom_input"
```
and the output be like:
{
    "subject_consistency": [
        0.0,
        [
            {
                "video_path": xxx,
                "video_results": 0.0
            },
            {
                "video_path":xxx,
                "video_results": 0.0
            },
            {
                "video_path": xxx,
                "video_results": 0.0
            },

same problem here https://github.com/Vchitect/VBench/issues/87

> same problem here #87
I track the bug found a strange thing is that, the image_features of the clip is all 0, which leads to the result of compute_background_consistency be zero. I don't know what's the problem, maybe the version of package?


Can you provide `ls -lh ~/.cache/vbench/*` and version here? Seems like the problem is not specific to CLIP as subject_consistency does not use it (https://github.com/Vchitect/VBench/issues/87). 

> > same problem here #87
> I track the bug found a strange thing is that, the image_features of the clip is all 0, which leads to the result of compute_background_consistency be zero. I don't know what's the problem, maybe the version of package?

I encountered the same problem and found the same thing as you. Besides, I noticed that the first model inference process took a unusually long time(approximately 10-12minutes) while other steps could finish in just minutes. Have you solved this bug?
Repository: Vchitect/VBench
Issue: Questions about the input prompts used in CogVideoX-5B (SAT prompt-optimized)
Root cause: Documentation Deficiency

When evaluating,  Prompts used in CogVideoX-5B (SAT prompt-optimized) is optimized? Can you provide the prompt used in CogVideoX's evalutation?

Yes, we applied the prompt optimisation pipeline according to CogVideoX's instructions. More details and the optimized prompt list can be found here: https://github.com/Vchitect/VBench/tree/master/prompts/gpt_enhanced_prompts

@ziqihuangg  thanks for your promply reply！another question
The optimzied prompts are only used in generating videos OR used in both generate and calculate vbench score?

Optimised prompts are only used to sample videos. The VBench evaluation still uses the original prompt before optimization.
Repository: Vchitect/VBench
Issue: how to calculate the total score of all metircs
GitHub labels: enhancement

thank you very much!

Thank you for your interest in VBench. You can refer to the calculation scheme at [Get Total Score](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard/blob/72b2069d0022f09f270393139dd39a86c57250cd/app.py#L82-L88) to calculate the standardized scores and weighted scores of different dimensions, obtaining quality scores and semantic scores. Then, sum them up to get the total score. 

Alternatively, you can upload evaluation results for all dimensions, which will be calculated by the Leaderboard. We will add this feature directly into the code of VBench as soon as possible.

ok，thanks


> Thank you for your interest in VBench. You can refer to the calculation scheme at [Get Total Score](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard/blob/72b2069d0022f09f270393139dd39a86c57250cd/app.py#L82-L88) to calculate the standardized scores and weighted scores of different dimensions, obtaining quality scores and semantic scores. Then, sum them up to get the total score.
> 
> Alternatively, you can upload evaluation results for all dimensions, which will be calculated by the Leaderboard. We will add this feature directly into the code of VBench as soon as possible.

Hi, do you add this feature in the code now？ Thanks
Repository: Vchitect/VBench
Issue: PyPI Versions and Compatibility with Torch
Root cause: Configuration Error


We address some common questions recently shared by our users (thanks for the feedback!). A fix will be included in the upcoming `vbench 0.1.5` release.

**Q1**: Why does `pip install vbench` install version `0.1.0` instead of `0.1.4`?
**A1**: That's probably because `torch 2.0.0` (or above) is already installed in your environment. Only `vbench 0.1.0` allows `torch 2.0.0 and above`.     `vbench 0.1.1` to `vbench 0.1.4` requires `torch<2.0.0 and >=1.12`. This strict dependency on Torch version is to ensure reproducibility of numerical results in some scenarios. We will try to relax this restriction to improve compatibility with newer Torch versions, and provide the fix in `vbench 0.1.5`.

**Q2**: When using `vbench` installed via pip, why do I encounter errors like `unrecognized arguments: --mode=custom_input`?
**A2**: That's because the vbench package is not up-to-date (see Q1 above for why).
Repository: Vchitect/VBench
Issue: Failed to install vbench
Root cause: External Dependency Breakage

I create a virtual environment via conda.
Then I pip vbench but failed with the following output.
![image](https://github.com/user-attachments/assets/778c589f-a43c-4cc4-bf56-96d6c04c4f43)
I had tried to adjust the version of transformers, but still failed.

Can you try the newer version of vbench? It should work with transformers==4.33.2 and pyiqa==0.1.10.

I tried to install a newer version by `pip vbench==0.1.4`, but it still failed.
![image](https://github.com/user-attachments/assets/b0d646b5-d5b8-4871-ac79-33954883fb79)
Repository: Vchitect/VBench
Issue: The number of samples

Thank you very much for this great work. I have a question: to evaluate on "temporal_flickering" dimension, how many samples do I need for each prompt? 40 or 5? 
```
for i, prompt in enumerate(longer_prompt_list):

       original_prompt = original_prompt_list[i]
       samples_per_prompt = 40 if dimension=="temporal_flickering" else 5

       for ind in range(samples_per_prompt):
           print(f"Sampling {prompt} ...")

```

For temporal flickering, we recommend sampling more than five videos per prompt (e.g., 40), and apply [static filtering](https://github.com/Vchitect/VBench?tab=readme-ov-file#before-evaluating-the-temporal-flickering-dimension-it-is-necessary-to-filter-out-the-static-videos-first). From the filtered results, keep the first 5 filtered static videos for each prompt. The exact number of samples required may vary depending on the characteristics of the video generative model.
Repository: Vchitect/VBench
Issue: Bug on vbench2_beta_long

Thank you for your great work. However I met some problems when using vbench2_beta_long:
```
 File "eval_long.py", line 283, in <module>
    main()
  File "eval_long.py", line 269, in main
    my_VBench.evaluate(
  File "/home/.conda/envs/vbench/lib/python3.8/site-packages/vbench2_beta_long/__init__.py", line 103, in evaluate
    self.preprocess(videos_path, mode, **kwargs)
  File "/home/.conda/envs/vbench/lib/python3.8/site-packages/vbench2_beta_long/__init__.py", line 49, in preprocess
    dimension_clip_length = load_clip_lengths(dimension_clip_length_config_path)
  File "/home/aiscuser/.conda/envs/vbench/lib/python3.8/site-packages/vbench2_beta_long/utils.py", line 438, in load_clip_lengths
    with open(yaml_file, 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/.conda/envs/vbench/lib/python3.8/site-packages/vbench2_beta_long/configs/clip_length_mix.yaml'
```
I think this may be due to the code, not sure:
```
dimension_clip_length_config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "configs", kwargs['clip_length_config'])
```
How should I solve this problem?


Have you solved it?
Repository: Vchitect/VBench
Issue: May I ask how to upload a video?

Our model is currently not open source. We hope to use the prompt you provided to infer the video results locally and upload the compressed video collection (. zip) for your evaluation.
Where can I upload video compression files?

You can share the videos with us via any cloud drive that is convenient for you.

> You can share the videos with us via any cloud drive that is convenient for you.

May I ask if it was sent to you via email?


Please find more instructions for submission here: https://github.com/Vchitect/VBench/blob/master/README-FAQ.md Thanks!
Repository: Vchitect/VBench
Issue: Question about Leadboard
Root cause: Documentation Deficiency

Thank you for you work!  We found that many numbers on the two leadboards "vbench" and "vbench long" are exactly the same. Shouldn’t the samples tested on these two leadboards be different?


It's the same set of videos if a model appears on both sub-leaderboards. Use [VBench](https://github.com/Vchitect/VBench?tab=readme-ov-file#usage) for evaluating videos shorter than 5.0 seconds (< 5.0s). Use [VBench-Long](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_long) for evaluating videos 5.0 seconds or longer (≥ 5.0s), and these models appear on both VBench and VBench-Long sub-leaderboards. Each benchmark is optimized for its respective video length to ensure fair and consistent evaluation.
Repository: allenai/reward-bench
Issue: adding kto as a separate category
Root cause: Documentation Deficiency

can KTO be added as a separate model type on the leaderboard?

My take is that because it's run exactly the same was as a DPO model, it's worth adding to the documentation but not sure it's worth the difference yet. If they start to diverge, we can revisit it in the future. 

the implied reward is upstream of the DPO objective, right? like you can calculate this with any pair of {base, finetuned/aligned models}, even ones finetuned with just SFT, and it should work (maybe not as well)

i would call it "implied RLHF reward" or something, since technically it predates the dpo work

Interesting, do you have a link @kawine -- i'd like to read some more of this history? But yeah, it's colloquial recognized as DPO but it seems like re-writing history then.

i think the DPO paper cites a few different papers as its inspiration in this respect, but the most clear precedent imo was set in a paper by Korbak et al (thm 1): https://arxiv.org/abs/2206.00761
Repository: allenai/reward-bench
Issue: [Model Request] mightbe/Better-PairRM
GitHub labels: New Model

https://huggingface.co/mightbe/Better-PairRM

I made few fixes on PairRM's dataset filter process and truncate etc..

As a result I got at least 15% performance.

Code and prompt template on huggingface repo.

Thank you.

@StableFluffy really cool. Do you have time to open a PR that extends the existing code in `models/pairrm.py` or make sure inference works? Looks like it shouldn't be too bad.

Closed with #104
Repository: allenai/reward-bench
Issue: Is eval set on huggingface the eval set or train set?
Root cause: Documentation Deficiency

Hi @natolambert et al,

We are reading the paper and the 2.98K filtered dataset at huggingface.

![Screenshot 2024-04-13 at 9 45 51 PM](https://github.com/allenai/reward-bench/assets/31041391/0d7165af-fc1b-4edc-9753-efbb3b92d759)

https://huggingface.co/datasets/allenai/reward-bench

I am curious if the huggingface 2.98K filtered data is the actual evaluation data used to evaluate on the leaderboard? 

Cause I looked into the code and saw this line in `utils.py`.

```
CORE_EVAL_SET = "ai2-adapt-dev/rm-benchmark-dev"
EXTRA_PREF_SETS = "allenai/pref-test-sets"
```

When I went to ai2-adapt-dev, I saw that it is a private dataset.

Asking cause we're hoping to know if we can/should train on the huggingface dataset for our reward model to fairly compare on the leaderboard.

Thanks!

Ah, @andrewsiah it's all one evaluation set (no training split). Yeah, missed this line, the final dataset is here. https://huggingface.co/datasets/allenai/reward-bench
That link was just a temporary home. Will update it now.
Repository: allenai/reward-bench
Issue: Best of N benchmark
Root cause: Unimplemented Feature Gap

1. Take a few chat models as the “base set”, say 1-3, like tulu 2 7b and tulu 2 13b (maybe olmo-instruct)
2. Generate ~8 completions per prompt in AlpacaEval (this is the heldout set)
3. Use each RM to choose the best-of-1 from that set, then run alpaca eval on the outputs
4. Score the delta for each RM in the batch on a set task (alpacaeval) and set base model (tulu)
5. Could do this with MTBench, but two turn is harder

Obvi flaws, but that seems WAY better than nothing.

@yuchenlin is starting this, woohoo!

Partially closed in #30 , wrapping up soon.
Repository: allenai/reward-bench
Issue: Clarification Needed on DPO Reward Evaluation
Root cause: Documentation Deficiency

Thank you for providing such a valuable benchmark.

I am seeking clarification on the model/reference specifications for DPO rewards, which are not readily apparent in either the paper or the leaderboard. For example, it is unclear whether models like Llama-3-8B-Instruct, Qwen, and Zephyr were evaluated without or with references.  If references were used, could you please provide guidance on how to access the reference model?

Thank you for your assistance.

Hey! @ZHZisZZ, good point, let's improve the docs. I committed an improvement to the leaderboard just now.
TLDR:
1. All models on the leaderboard are computed with a reference model. Reference free results are currently only in a coming version of the paper (not even updated on arxiv yet).
2. The reference models are all in the [reward bench results repo](https://huggingface.co/datasets/allenai/reward-bench-results). Specially, you must click on a specific model and you'll see a key `ref_model`, e.g. [Qwen](https://huggingface.co/datasets/allenai/reward-bench-results/blob/main/eval-set/Qwen/Qwen1.5-72B-Chat.json).  Alternatively, all the reference models are listed in the [evaluation configs](https://github.com/allenai/reward-bench/blob/main/scripts/configs/eval_configs.yaml).

If you go to a [folder](https://huggingface.co/datasets/allenai/reward-bench-results/tree/main/eval-set/Qwen), you can see the ref free results are separate.

The reference free results can also be loaded with this script https://github.com/allenai/reward-bench/blob/main/analysis/get_dpo_ref_free_results.py

Thank you for your clarifications. I figured out that some reference models are pre-trained models, which may not be the actual reference models that were used for training. This can result in some issues because these pre-trained models do not always recognize the prompt template for the fine-tuned models.   

And, out of curiosity, I have some extra questions:

1) Based on your experience, do these pre-trained reference models offer advantages over the reference-free version? 

2) If so, then probably we can use an arbitrary pre-trained model as a ref as long as it shares the same vocabulary as the fine-tuned model (although page 11 of the paper says using the wrong ref may lead to bad performances).



@ZHZisZZ I think it'll be a mixed bag. You'll probably gain 1-5% performance by using the SFT checkpoint vs the base model. Some models aren't documented well and may just not have an SFT checkpoint.

Swapping the wrong reference model we tried (accidentally) and performance entirely regressed to random on a few models.
Repository: allenai/reward-bench
Issue: Do we need to add system prompt when training/evaluating RM?
Root cause: Documentation Deficiency

Hi,

I want to know if we should add a system prompt when training and evaluating our reward models. For instance, I have supervised fine-tuned LLaMA2-7B on the Alpaca SFT 10k subset, and I use the same chat template for the tokenizer as the one in [LLaMA2-7B-Chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/f5db02db724555f92da89c216ac04704f23d4590/tokenizer_config.json#L12). The system prompt used in Meta's paper is shown in the screenshot below, which is included manually during my SFT procedure.

My understanding is that since we add a system prompt when collecting responses from LLaMA2 during RLHF, the system prompt becomes part of the overall prompt seen by LLaMA2. Therefore, it seems logical to include the system prompt in the query sent to the reward model to better assess the quality of the generated responses. However, if we apply the tokenizer's chat template, as is the default behavior in Reward-Bench, the system prompt might not be included.

Could you please clarify if the system prompt should be added during the training and evaluation of reward models?

![image](https://github.com/allenai/reward-bench/assets/55573473/2410c989-2606-4566-bcc0-8f49182bd8f2)

Thank you!

Hey @hank0316 to the best of my knowledge, no, but you can likely do it.
Reward models normally use the same chat template from IFT which allows it, but it's not well studied.

If it is added during training, it should be used during evaluation. Of all the models added, I don't remember ever seeing this mentioned in the model card. 

Please re-open the issue if you see a model with this!
Repository: allenai/reward-bench
Issue: possibly a typo in `load_bon_dataset.py`
Root cause: Algorithmic Error

Line 486 applies the map function to raw_dataset instead of the "processed" unrolled_dataset, it will produce a key error since raw_dataset does not have the key "input". If I switch to unrolled_dataset the code could continue as usual.

https://github.com/allenai/reward-bench/blob/e7b62d8c61e37567c26d52fce6beb9c52b4a8f42/rewardbench/utils.py#L476-L490


```
# raw_dataset
Dataset({
    features: ['dataset_details', 'output', 'model_input', 'model', 'id', 'config', 'prompt', 'subset'],
    num_rows: 1770
})
```

```
# unrolled_dataset
Dataset({
    features: ['dataset_details', 'model_input', 'model', 'config', 'prompt', 'subset', 'input', 'id'],
    num_rows: 7080
})
```
Repository: allenai/reward-bench
Issue: Model Test Application
Root cause: Validation Gap

Hello, my lab has recently fine-tuned a new model and I'd like to contribute my model to your leaderboard, please refer to https://huggingface.co/CIR-AMS/BTRM_Qwen2_7b_0613 for details

@wjxxyz can you modify the checkpoint so it can be loaded from HuggingFace without special code? Right now the file is nested in folders so it requires code modification / special handling.

> @wjxxyz can you modify the checkpoint so it can be loaded from HuggingFace without special code? Right now the file is nested in folders so it requires code modification / special handling.您可以修改检查点，以便无需特殊代码即可从 HuggingFace 加载它吗？现在，该文件嵌套在文件夹中，因此需要代码修改/特殊处理。

I've changed the checkpoints and the web link is https://huggingface.co/CIR-AMS/BTRM_Qwen2_7b_0613/tree/main, thank you

Thanks! @wjxxyz I also opened #155 to make bfloat16 work correctly.
Repository: allenai/reward-bench
Issue: Add New reward models 
Root cause: Unimplemented Feature Gap

Two new reward models are available: Ray2333/GRM-llama3-8B-distill (https://huggingface.co/Ray2333/GRM-llama3-8B-distill), Ray2333/Gemma-2B-rewardmodel-baseline (https://huggingface.co/Ray2333/Gemma-2B-rewardmodel-baseline). They are both finetuned on opensource datasets, achieving average scores of 86.1 and 73.7 locally. 

Details are on the huggingface pages.



Nice, will add these shortly with `Ray2333/GRM-Gemma-2B-sftreg` and `Ray2333/GRM-llama3-8B-sftreg`. Cool paper!

Thank you!
Repository: allenai/reward-bench
Issue: Add new Reward Model and Generative Model.
Root cause: Unimplemented Feature Gap

We have opened two models: one is a reward model, and the other is a generative model with a fixed prompt template like Prometheus.
Details are on the huggingface pages and our [paper](https://www.arxiv.org/pdf/2407.06551).

1. [NCSOFT/Llama-3-OffsetBias-RM-8B](https://huggingface.co/NCSOFT/Llama-3-OffsetBias-RM-8B)
2. [NCSOFT/Llama-3-OffsetBias-8B](https://huggingface.co/NCSOFT/Llama-3-OffsetBias-8B)

Our local result for the reward model is {'Chat': 0.9776536312849162, 'Chat Hard': 0.8201754385964912, 'Safety': 0.8839476307476307, 'Reasoning': 0.9244591313362799}.
And generative model result is {"Chat": 0.9413407821229051, "Chat Hard": 0.8004385964912281, "Safety": 0.8622770094770095, "Reasoning": 0.7627134828395264}.

For the reward model, it's okay to evaluate without any additional code, but for the generative model, may I open a pull request for our model?

Hey @sanghyuk-choi please open a pull request if it doesn't work with VLLM (our default local model tool)
https://github.com/allenai/reward-bench/blob/6a5e0c4315b84beccfbbd99ca4878fda8ae31d56/scripts/run_generative.py#L125

Thank you @natolambert . 
I've just opened pull requests for this issue. see https://github.com/allenai/reward-bench/pull/159
and I would greatly appreciate it if you add our two models to the reward bench.

Due to slight difference in prompt templates and dtype of models since I left first comment,
the [NCSOFT/Llama-3-OffsetBias-RM-8B](https://huggingface.co/NCSOFT/Llama-3-OffsetBias-RM-8B) (reward model) score would be {"Chat": 0.9720670391061452, "Chat Hard": 0.8070175438596491, "Safety": 0.890087328887329, "Reasoning": 0.9054468816500246}
and the [NCSOFT/Llama-3-OffsetBias-8B](https://huggingface.co/NCSOFT/Llama-3-OffsetBias-8B) (generative model) score would be {'Chat': 0.9217877094972067, 'Chat Hard': 0.8026315789473685, 'Safety': 0.8609156897156898, 'Reasoning': 0.7611856823266219}

you can use our default chat template.
Repository: allenai/reward-bench
Issue: ArmoRM Issue
Root cause: Documentation Deficiency

Running `rewardbench --model=RLHFlow/ArmoRM-Llama3-8B-v0.1` results in the following error:

```
Traceback (most recent call last):
  File "/miniconda/envs/reward-bench/bin/rewardbench", line 33, in <module>
    sys.exit(load_entry_point('rewardbench', 'console_scripts', 'rewardbench')())
  File "/mnt/bn/hongyi-lq/mlx/users/hongyi.guo1/repo/12252/reward-bench/rewardbench/rewardbench.py", line 142, in main
    raise NotImplementedError("Custom dialogue not implemented yet for simpler data formatting.")
NotImplementedError: Custom dialogue not implemented yet for simpler data formatting.
```

Could you please provide the correct way to evaluate ArmoRM?

Ah, yeah the CLI is for general datasets (which doesn't work for some architectures).
It is run with:
```
python scripts/run_rm.py --model= RLHFlow/ArmoRM-Llama3-8B-v0.1
```
Trying to keep the CLI scripts simpler was a decision, still open to other interpretations.

> Ah, yeah the CLI is for general datasets (which doesn't work for some architectures). It is run with:
> 
> ```
> python scripts/run_rm.py --model= RLHFlow/ArmoRM-Llama3-8B-v0.1
> ```
> 
> Trying to keep the CLI scripts simpler was a decision, still open to other interpretations.

Running with `python scripts/run_rm.py --model= RLHFlow/ArmoRM-Llama3-8B-v0.1`, but result following error: 
```
 raise ValueError(
ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode. Please rerun your script specifying `--num_processes=` or by launching with `python {{myscript.py}}`.
```



@t-sifanwu I believe this is because you're using multiple GPUs for inference. 
The simple implementation we used has some rough edges around this (such as this). I'm not sure at the `train` comment, but let me know if you need more help.