zhiminy/EvalEng huggingface dataset:
Repository: allenai/reward-bench
Issue: Add SmolTulu-1.7b-RM
Root cause: Interface Contract Mismatch

Doesn't require any custom code:

https://huggingface.co/SultanR/SmolTulu-1.7b-RM

Hi @SulRash— thanks for your patience as we catch up after the holidays! We ran the huggingface version of your model but aren't able to reproduce the scores from the model [card](https://huggingface.co/SultanR/SmolTulu-1.7b-RM#evaluation).

The model [config](https://huggingface.co/SultanR/SmolTulu-1.7b-RM/blob/main/config.json#:~:text=architectures%22%3A%20%5B-,%22LlamaForCausalLM%22,-%5D%2C) has the architecture `LlamaForCausalLM`, which seems inconsistent with [this line](https://huggingface.co/SultanR/SmolTulu-1.7b-RM#:~:text=model%20%3D%20AutoModelForSequenceClassification.from_pretrained(checkpoint).to(device)) from your example usage:
`model = AutoModelForSequenceClassification.from_pretrained(checkpoint).to(device)`.

Running the above and the rewardbench code both lead to the following warning:
`Some weights of LlamaForSequenceClassification were not initialized from the model checkpoint at SultanR/SmolTulu-1.7b-RM and are newly initialized: ['score.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.`

This mismatch in architecture is likely causing the issue with the scores, could you take a look at this and let us know?

Huh, what an oversight from my end, let me have a look at it tonight and I'll see if I can rectify this. Weirdly enough I managed to run rewardbench and the open-instruct rlvr just fine with the same model. Happy holidays!

@SulRash I was looking with her -- looks like the sort of thing that can happen where you reference the weights locally and then when uploading, the hf abstractions get weird and it mis-casts the config. 

Closing as stale.
Repository: allenai/reward-bench
Issue: Add new models
Root cause: Unimplemented Feature Gap

Hi @natolambert, 
can you please add these two models:
nicolinho/QRM-Gemma-2-27B
nicolinho/QRM-Llama3.1-8B-v2

They do not need custom code and can be run with the following commands:
`python scripts/run_rm.py --model nicolinho/QRM-Gemma-2-27B --batch_size=1 --torch_dtype=bfloat16  --attn_implementation=flash_attention_2 --max_length 4096   --not_quantized --trust_remote_code`
{'Chat': 0.9664804469273743, 'Chat Hard': 0.9013157894736842, 'Safety': 0.9283783783783783, 'Reasoning': 0.9836375293283135} Overall Score: 0.9449530360269376
`python scripts/run_rm.py --model nicolinho/QRM-Llama3.1-8B-v2 --batch_size=1 --torch_dtype=bfloat16  --attn_implementation=flash_attention_2   --not_quantized --trust_remote_code`
{'Chat': 0.9692737430167597, 'Chat Hard': 0.868421052631579, 'Safety': 0.927027027027027,  'Reasoning': 0.9670670322475038} Overall Score: 0.9329472137307174

Thank you!

@Nicolinho we're catching up on this after break. Thanks for patience.

Hi @Nicolinho — your models are now on the leaderboard! Great work!
Repository: allenai/reward-bench
Issue: [BUG]ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. 
Root cause: External Dependency Breakage


Hello, when I run the following command:

```
python scripts/run_rm.py --model=openbmb/Eurus-RM-7b --chat_template=openbmb
```

I always get an error.
![image](https://github.com/user-attachments/assets/d5c36938-4bc4-4177-9d63-aa0a4aff2d56)


Hi @noforit — thanks for pointing this out! This error gets resolved with the updated version of transformers (we have updated setup.py to pin to the current version 4.47.1 now). However, please note that you should run `openbmb/Eurus-RM-7b` with `trust_remote_code=True`. With the updated transformers version, the following command works:
`python scripts/run_rm.py --model=openbmb/Eurus-RM-7b --chat_template=openbmb --trust_remote_code`

Thanks for your patience as we catch up after the holidays!

Another solution is just to set `--torch_dtype=bfloat16` to disable quantization, as the defaulted `float16` will always trigger quantization.
Repository: allenai/reward-bench
Issue: Add new generative models
Root cause: Unimplemented Feature Gap

We would like to add  RISE-Judge-Qwen2.5-32B and RISE-Judge-Qwen2.5-7B by:

`python scripts/run_generative.py --model="R-I-S-E/RISE-Judge-Qwen2.5-32B"`

and

`python scripts/run_generative.py --model="R-I-S-E/RISE-Judge-Qwen2.5-7B"`

Different with the default parameters, we set vllm output maxtokens to 4096.

Our support code can be seen in pull request add RISE support code #217 

This is our local test results:

RISE-Judge-Qwen2.5-32B:

{"Chat": 0.9664804469273743, "Chat Hard": 0.8333333333333334, "Safety": 0.918918918918919, "Reasoning": 0.98770256997872}

RISE-Judge-Qwen2.5-7B:

{"Chat": 0.9217877094972067, "Chat Hard": 0.7653508771929824, "Safety": 0.8797297297297297, "Reasoning": 0.9607716756697768}

Thanks!
Repository: allenai/reward-bench
Issue: Unpin transformers version in setup.py
Root cause: Configuration Error

In setup.py, the transformers package is currently pinned to version 4.47.1 with the comment: `# pinned at most recent version`

However, newer versions of transformers have been released since then. Keeping it pinned to 4.47.1 causes compatibility issues when installing rewardbench into existing environments.

I tested removing the version pin, allowing pip to install the latest transformers version, and everything worked smoothly.

Is there a specific reason for pinning transformers at 4.47.1? If not, would it be possible to unpin or use a more flexible version constraint (e.g., transformers>=4.47.1)?

@saumyamalik -- thoughts? In general, we should ALWAYS pin huggingface libraries -- else random things may break or change, but updating is fine when we test it. 

Hi @MariSchn — we pin libraries to ensure compatibility, and the current set of versions work. We will look into pinning to a more recent version, but will spend some time testing it first.
Repository: allenai/reward-bench
Issue: Chat template for Llama3-based models
Root cause: Configuration Error

I'm trying to reproduce the accuracy numbers for a few models from the leaderboard and they are very far off. I suspect that I'm not using the right chat template. The chat template is pulled from fastchat, but its last released version 0.2.36 doesn't support the Llama 3 chat template. So which template should be specified when running `run_rm.py` for models like infly/INF-ORM-Llama3.1-70B? Not specifying the `chat_template` argument just defaults it to "tulu".

I was able to reproduce the results for some of the Llama3-based models by patching the Llama3 chat template into fastchat locally (following PR https://github.com/lm-sys/FastChat/pull/3259). But this still makes me wonder how RewardBench works with the last released version of fastchat without Llama3 support?

@alexnikulkov if there's a chat template in the tokenizer reward bench should use that? Which, for this model is here: https://huggingface.co/infly/INF-ORM-Llama3.1-70B/blob/4f26af90d5185719879de65502ed4b098f993a9c/tokenizer_config.json#L2053

Could it be something with the the custom code? https://huggingface.co/infly/INF-ORM-Llama3.1-70B/blob/4f26af90d5185719879de65502ed4b098f993a9c/config.json#L3

Implementation in rewardbench we didnt look super closely into, was made by the model creators? https://github.com/allenai/reward-bench/blob/main/rewardbench/models/inform.py

Hmm, I re-tested on "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2" and got matching results even without the Llama3 template from fastchat. Looks like the `--chat_template` argument is just getting ignored and chat formatting is being done by the HF tokenizer. There might be an opportunity to clean up the code by removing `get_conv_template()` from `run_rm.py` if it's not being used.
Repository: allenai/reward-bench
Issue: Add new multimodal reward model
Root cause: Unimplemented Feature Gap

env
transformers==4.51.3
trl==0.16.1
torch==2.6.0
torchvision==0.21.0
accelerate==1.6.0

We would like to add a multimodal reward model [Skywork/Skywork-VL-Reward-7B](https://huggingface.co/Skywork/Skywork-VL-Reward-7B) by:

python scripts/run_rm.py --model Skywork/Skywork-VL-Reward-7B --trust_remote_code --torch_dtype bfloat16 --batch_size 1

This is a multimodal reward model that shows promising and competitive performance on RewardBench.

Our support code can be seen in https://github.com/allenai/reward-bench/pull/229

This is our local test results:
{'Chat': 0.8994413407821229, 'Chat Hard': 0.875, 'Safety': 0.9108108108108108, 'Reasoning': 0.9175567468761936}

Thanks!

<img width="1471" alt="Image" src="https://github.com/user-attachments/assets/00ce46d6-9b56-4da1-8c8b-f6b162cceaca" />
@shawn0wang -- results are live. We'll get inference support merged shortly.

Hi @shawn0wang just to update you— inference support is merged, and the model is also [live](https://huggingface.co/spaces/allenai/reward-bench) on RewardBench v2! Great work!
We ran it with:
```
python scripts/run_v2.py --model Skywork/Skywork-VL-Reward-7B --tokenizer Skywork/Skywork-VL-Reward-7B --batch_size 1 --trust_remote_code --torch_dtype=bfloat16
```

<img width="1145" alt="Image" src="https://github.com/user-attachments/assets/3a1e8925-ecef-44d3-9925-850eaacdb50b" />
Repository: allenai/reward-bench
Issue: Add a new reward model (RAMO)
Root cause: Unimplemented Feature Gap

We provide an 8B reward model:
      https://huggingface.co/HFXM/RAMO-Llama3.1-8B

python reward-bench/scripts/run_rm.py \
  --model=HFXM/RAMO-Llama3.1-8B  \
  --chat_template=Ziya \
  --batch_size=8

Local running results:
Chat: 92.2
Chat Hard: 89.0
Safety: 95.1
Reasoning: 93.5
Overall: 91.9

Hey @mingye1101 sure, but we're releasing v2 soon-ish and may be best to just wait and add to that :D

Hi @mingye1101, we added your model to v2! Nice job! Just a heads up— we found much stronger performance with using your model's tokenizer rather than the Ziya chat template, so we kept those numbers for the leaderboard (as is, our inference code  makes a usable tokenizer take precedence over a chat template, so it is likely your local run for v1 also used the tokenizer). Let us know if you have any issues with that.

<img width="1162" alt="Image" src="https://github.com/user-attachments/assets/337bc524-4eeb-4d22-b15f-9869e6d35ea3" />
Repository: allenai/reward-bench
Issue: Fail to Run a Model on RewardBench v2
Root cause: Documentation Deficiency

Precise IF: 0/160 (0.0)
Filter: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1865/1865 [00:00<00:00, 59592.71 examples/s]
Safety: 0/450 (0.0)
Filter: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1865/1865 [00:00<00:00, 56640.38 examples/s]
Traceback (most recent call last):
  File "/mnt/vast/home/jimmy/evaluation/reward-bench/scripts/run_v2.py", line 387, in <module>
    main()
  File "/mnt/vast/home/jimmy/evaluation/reward-bench/scripts/run_v2.py", line 328, in main
    ties_subset_with_results, overall_score = process_single_model(subset_dataset)
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/reward-bench/rewardbench/utils.py", line 1064, in process_single_model
    k: {prompt_id: sample_stats(samples) for prompt_id, samples in type_results.items()}
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/reward-bench/rewardbench/utils.py", line 1018, in sample_stats
    best_incorrect = max(incorrect_samples, key=lambda x: x["scores"][0])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'

The previous error is meet when running the following code. Would you mind helping me figure out the problem.

python scripts/run_v2.py --model=nicolinho/QRM-Gemma-2-27B --dataset=allenai/reward-bench-2 --batch_size=8

@natolambert Thank you for your help!


ok, i met same problem. i fix it by change torch_dtype to bfloat16.

@Zhichao-Jimmy970201 @wutong4012 we've had some issues with Gemma, particularly due to a different attention implementation. I had headaches with this for v1, regardless @saumyamalik is going to take another look soon iirc. It was already on our radar!

Same for #240 iirc, or at least similar.

Hi @Zhichao-Jimmy970201 and @wutong4012! Thanks for using RewardBench 2 — indeed, the issue you're seeing is because Gemma 2 models require flash attention 2 (and also trust_remote_code=True). nicolinho/QRM-Gemma-2-27B should be run with the following command:

`scripts/run_v2.py --model nicolinho/QRM-Gemma-2-27B --dataset allenai/reward-bench-2 --tokenizer nicolinho/QRM-Gemma-2-27B --batch_size 1 --trust_remote_code --torch_dtype=bfloat16 --attn_implementation flash_attention_2`

In particular, note the `trust_remote_code`, `torch_dtype`, and `attn_implementation` arguments that are crucial for performance. Finally, make sure that your flash attention version==2.6.3 as specified in the [Dockerfile](https://github.com/allenai/reward-bench/blob/main/Dockerfile) — we observed low performance with old versions.

These additional arguments can be found in [eval configs](https://github.com/allenai/reward-bench/blob/main/scripts/configs/eval_configs.yaml) `scripts/configs/eval_configs.yaml`— we will add an explicit link to these configs to the readme for clarity!

Let us know if you have any more issues!
Repository: allenai/reward-bench
Issue: Experiment with human vs gpt4 data
Root cause: Unimplemented Feature Gap

With the human data AI2 has or a dataset like `no_robots`, we could test if a RM prefers the human or model answers to a completion.


Update: should use an open weights model for completions to private prompts, otherwise company with API has access to closed test set prompts.

Closing as stale.
Repository: allenai/reward-bench
Issue: Some error & fix in RewardBench2
Root cause: Validation Gap

Hello, thank you for your contribution to the open source community. I encountered some bugs when reproducing RewardBench2.
Specifically, group['scores'] here is two-dimensional, as shown in the following figure:

![Image](https://github.com/user-attachments/assets/da5ed8d9-5082-49e8-b561-ed58c51a41c7)

If it is just tolist(), it is a List containing array,

![Image](https://github.com/user-attachments/assets/df560deb-f2a0-4e31-b2cb-24fc07389564)

This will cause an error here:

https://github.com/allenai/reward-bench/blob/ece3d671c741e540be37a94ef918db704b063b31/rewardbench/utils.py#L562

To solve this problem, I flattened group[col] completely and turned it into a one-dimensional List, but I don't know if this is correct.

`group[col].explode().tolist()`

Looking forward to your reply.

顺便说一下，我复现的Skywork/Skywork-Reward-Gemma-2-27B-v0.2的结果很低，请问有哪些可能导致这样的问题呢？

我在A100-80G上的运行命令为
`python scripts/run_v2.py --model Skywork-Reward-Gemma-2-27B-v0.2 --torch_dtype bfloat16 --batch_size 4`

得到的结果为
  | Factuality | Precise IF | Math | Safety | Focus | Ties | Overall
-- | -- | -- | -- | -- | -- | -- | --
Skywork-Reward-Gemma-2-27B-v0.2 | 52.42 | 28.75 | 38.52 | 88.89 | 67.47 | 63.29 | 56.56

Hi @wutong4012 — thanks for using RewardBench 2! 

On your first question— which model are you using? The pipelines in the repo all output one score (most models specify how to reduce multiple outputs to one score), so this issue doesn't arise for any of the models we evaluated. 

As for your second question, make sure you are running with flash attention version 2.6.3, specified in the [Dockerfile](https://github.com/allenai/reward-bench/blob/main/Dockerfile)! We also observed weirdly lower performance with older versions. In particular, I was able to reproduce your lower numbers with version 2.5.0, so that could be the culprit. Let us know if that doesn't fix it!

Also, you should refer to the [eval configs](https://github.com/allenai/reward-bench/blob/main/scripts/configs/eval_configs.yaml) `scripts/configs/eval_configs.yaml` for model-specific arguments that should be specified in the command (we will add a reference to this file in the readme for added clarity!). For example, Skywork/Skywork-Reward-Gemma-2-27B-v0.2 should be run with the following command:
`python scripts/run_v2.py --model Skywork/Skywork-Reward-Gemma-2-27B-v0.2 --tokenizer Skywork/Skywork-Reward-Gemma-2-27B-v0.2 --batch_size 2 --torch_dtype=bfloat16 --attn_implementation flash_attention_2`

(Note that Skywork/Skywork-Reward-Gemma-2-27B-v0.2 requires the `--attn_implementation flash_attention_2` argument)

Thanks for your reply. I changed to flash_attention_2 and successfully reproduced the result.
Regarding the first question, I don't know what went wrong at that time. Now there is no problem. Let it go.
Repository: allenai/reward-bench
Issue: local variable 'results_ties' where it is not associated with a value
Root cause: Validation Gap

https://github.com/allenai/reward-bench/blob/5822f16d49e187d04ad7404a8ed9e232c68f0e05/scripts/run_generative_v2.py#L471

The results_ties variable seems to be defined only when using the api. This error will be reported when using force_local.
But why does ties need to be handled separately? If so, maybe this should also be handled separately.

While running, I found some typos and submitted PR(https://github.com/allenai/reward-bench/pull/247) to fix them.

Hi @wutong4012 — we are actively working on adding support for vllm models, will address this either today or tomorrow!

Hi @wutong4012 — this should be fixed now! Feel free to give it a go!
Repository: allenai/reward-bench
Issue: RewardBench2 Submission with API models
Root cause: Documentation Deficiency

HI @saumyamalik @natolambert , I wanted to ask what the procedure would be to make a submission of an API model on RewardBench2. Thanks

Ah, if the model is behind a private API we generally have done email us the results json and we manually add it on the back end. If this is going to be a public API, we can try to refactor the code to support it. Let us know @williamberrios.

Mostly, if you want to get scores on the leaderboard, email us (or paste here if you don't care about confidentiality). If you want support for people to run the model, opening a PR is best.

Oh nice! Thank you @natolambert 🙌 . I'll check back with the team and decide what to do here. Also, could you share the email? 

http://natolambert.com/contact ;) try not to paste it more places that always get scraped
Repository: allenai/reward-bench
Issue: Add Unsloth Fast Inference option
Root cause: Unimplemented Feature Gap

See https://github.com/vllm-project/vllm/issues/8700#issuecomment-2964602903

Can be up to 2x+ speedup.

Example: https://gist.github.com/corbt/25eb21d47bbd2add810f1cf0edeea1b6
Repository: allenai/reward-bench
Issue: Evaluation gemma-2-27b-it
Root cause: Validation Gap

Hi, have you tried to evaluate gemma-2-27b-it using generative methods?
Gemma seems to be the most used and has the best results, but it seems to have many details.

I tested it and found that template would report an error, which can be solved by deleting "system". Then run it and get all 0.5 results (as shown below).
![Image](https://github.com/user-attachments/assets/863658b4-74fa-4a5f-b7dd-b8b88dbf9550)

In fact, I am not sure whether this result is correct (it is not intuitively correct). According to previous experience, bf16 and flash_attn should be used. Then I set dtype to bf16 when initializing LLM, and set VLLM_ATTENTION_BACKEND=FLASH_ATTN, but the results are still all 0.5.

Do you have any solution to this? Or is the result of the basic model actually like this?

Hi @wutong4012 — the result is indeed not correct (the generative script deals with errors, parsing and otherwise, by defaulting to a random baseline of 0.5, which is likely what you are observing here). We will look into this soon and get back to you! Thanks!
Repository: allenai/reward-bench
Issue: Reward Bench2 Metric not consistent to Datasets: allenai/reward-bench-2-results
Root cause: Validation Gap

Hi, I'm trying to calculate the metric of `google/gemini-2.5-flash-preview-04-17` by loading` allenai/reward-bench-2-results` Datasets on Huggingface, but cannot get the same result as the metric on RewardBench2 LeaderBoard.

Could any one kindly point out the reason or any mistake ❤️?

<img width="1139" height="323" alt="Image" src="https://github.com/user-attachments/assets/5d00969b-5736-4c68-8a73-b21387d75f23" />

```
from datasets import load_dataset

google_dataset = load_dataset("allenai/reward-bench-2-results", data_files="eval-set-scores/google/**")
num_correct, num_total = 0, 0

for i, example in enumerate(google_dataset['train']):
    if example['model'] != "google/gemini-2.5-flash-preview-04-17":
        continue
    if example['subset'] != 'Math': # 81.1
      continue
    num_correct += example['results']
    num_total += 1

print(num_correct / num_total)
print(num_correct, num_total)
```

<img width="1647" height="444" alt="Image" src="https://github.com/user-attachments/assets/bcda3e4b-f2c4-49d3-b36e-e337ed2fe6c6" />
Repository: allenai/reward-bench
Issue: Add SARM
Root cause: Unimplemented Feature Gap

-----

### **Title: Add SARM-4B to RewardBench v2 Leaderboard**

Hi team,

We would like to submit our new reward model, **SARM-4B**, for inclusion in the RewardBench v2 leaderboard. Below are the details of our model and our local evaluation results.

-----

### **Model Information**

  * **Model Name**: `Llama-SARM-4B`
  * **Paper**: [[SARM: Interpretable Reward Model via Sparse Autoencoder](https://arxiv.org/abs/2508.08746)](https://arxiv.org/abs/2508.08746)
  * **Model Card**: [[schrieffer/SARM-4B](https://www.google.com/search?q=https://huggingface.co/schrieffer/SARM-4B)](https://www.google.com/search?q=https://huggingface.co/schrieffer/SARM-4B)
  * **Base Model**: Finetuned from `meta-llama/Llama-3.1-8B-Instruct`
  * **Code Repository**: [https://github.com/schrieffer-z/sarm](https://github.com/schrieffer-z/sarm)
  * **Demo**: [[SARM Demo on Hugging Face Spaces](https://huggingface.co/spaces/Schrieffer/SARM-Demo)](https://huggingface.co/spaces/Schrieffer/SARM-Demo)

-----

### **Evaluation Details**

We ran the evaluation locally on RewardBench v2 using the following command:

```shell
python scripts/run_v2.py \
  --model Schrieffer/Llama-SARM-4B \
  --batch_size 8 \
  --torch_dtype bfloat16 \
  --trust_remote_code
```

Here are our local results:

```json
{
  "Factuality": 0.6873684210526316,
  "Focus": 0.9555555555555556,
  "Math": 0.644808743169399,
  "Precise IF": 0.428125,
  "Safety": 0.9177777777777778,
  "Ties": 0.7939410374302791,
  "Average": 0.7379294224976073
}
```
-----
Since there is a sae decoder in our model which is not needed for reward model inference just for convenience and further interpretability, I would like to use 4.3B (4.02B backbone +0.27B sae encoder) as model size. 
Here is module sizes of components of Llama-SARM-4B:
sae decoder(0.27B)
sae encoder(0.27B)
backbone （4.02B）

Thank you for your time and consideration. Please let us know if you need any further information.


Hi, thank you for using RewardBench 2! Apologies for the delay, but the model is up on the [leaderboard](https://huggingface.co/spaces/allenai/reward-bench). Congrats on the great work!
Repository: allenai/reward-bench
Issue: GRM inference on RewardBench breaks with transformers>=4.51.0
Root cause: External Dependency Breakage

I trained a GRM model using the [official GRM code](https://github.com/YangRui2015/Generalizable-Reward-Model) and then ran inference on RewardBench.
When using transformers==4.51.0, the rewards produced during inference are almost all None. However, when using an earlier version, transformers==4.45.2, the rewards are computed normally.

I've resolved this issue: the NaN rewards in transformers==4.45.2 stem from numerical instability. 
Adding `torch_dtype=torch.float32` when loading the model in `GRewardModel.__init__` fixed it, and rewards now compute normally.