zhiminy/EvalEng huggingface dataset:
Repository: allenai/reward-bench
Issue: Add Cloud RMs
Root cause: Unimplemented Feature Gap

See @zankner's repo https://github.com/zankner/CLoud, RM's that think out loud!

Hi @natolambert, I recently trained a generative RM ([prometheus-eval/prometheus-RM-Llama-8B-v1.0](https://huggingface.co/prometheus-eval/prometheus-RM-Llama-8B-v1.0)) based on the CLoud code base and l think the inference using the huggingface transformers can be easily integrated to existing code of reward-bench. Can I try working on this?

@scottsuk0306 yes please! I've been curious for a bit and we're building new datasets now

FYI -- my issue was the tokenization timing was specific and would require a bunch of handling or a refactor. Lmk if you figure out something clever.

Hi sorry realized I never followed up on this. @scottsuk0306 @natolambert we support HF inference in the CLoud repo which I can also add ([hf-inference](https://github.com/zankner/CLoud/blob/fac417e2c0f45fab9083bf9b77066de490e8288c/cloud/model.py#L105)). However, HF generate is extremely slow. More recently we added [vllm-support](https://github.com/zankner/CLoud/blob/main/cloud/inference/vllm_llama_cloud.py) but I'm not sure whether that would be easy to support in your repo, if you have any thoughts.

We could try running Cloud RM with the "generative" pipeline, which is a bit different.
Otherwise, I feel like it isn't needed to use VLLM, unless we can figure out how to wrap it in the same abstraction -- which actually may be possible, because we just do some general model loading.

Either way, curious where we end up.

Closing as stale.
Repository: allenai/reward-bench
Issue: Add new reward models
Root cause: Unimplemented Feature Gap

Hi, thanks for your work :)

We have uploaded two new RMs:

1. [LxzGordon/URM-LLaMa-3.1-8B](https://huggingface.co/LxzGordon/URM-LLaMa-3.1-8B)
2. [LxzGordon/URM-LLaMa-3-8B](https://huggingface.co/LxzGordon/URM-LLaMa-3-8B)

The models can be evaluated with no additional code. We locally evaluated the RMs and the results are:

```bash
python scripts/run_rm.py  --model LxzGordon/URM-LLaMa-3.1-8B --trust_remote_code --not_quantized
# {'Chat': 0.952513966480447, 'Chat Hard': 0.8837719298245614, 'Safety': 0.9108108108108108, 'Reasoning': 0.969304168712828}
```
```bash
python scripts/run_rm.py  --model LxzGordon/URM-LLaMa-3-8B --trust_remote_code --not_quantized
# {'Chat': 0.9692737430167597, 'Chat Hard': 0.7872807017543859, 'Safety': 0.8837837837837837, 'Reasoning': 0.9584220003273858}
```

Could you please add these two models to the leaderboard? Very much thanks!
Repository: allenai/reward-bench
Issue: Add A New Generative Model
Root cause: Unimplemented Feature Gap

Hi RewardBench Team,

We have updated a new generative model:
- [Skywork/Skywork-Critic-Llama-3.1-8B](https://huggingface.co/Skywork/Skywork-Critic-Llama-3.1-8B)

Our local evaluation metrics for the model is listed as bellow:
```bash
# Skywork-Critic-Llama-3.1-8B: 
{
    'Chat': 0.9385474860335196, 
    'Chat Hard': 0.8135964912280702, 
    'Safety': 0.9159293787293787, 
    'Reasoning': 0.8975350575653408
}
```

For the generative model, it's okay to evaluate it with the default `scripts/run_generative.py` script.
```
cd reward-bench
model_name_or_path="Skywork/Skywork-Critic-Llama-3.1-8B"
python scripts/run_generative.py --model $model_name_or_path --trust_remote_code --do_not_save --force_local --num_gpus 1 2>&1 | tee ./evaluation_logs.txt
```

We would like to add this new generative model to the RewardBench LeaderBoard.

Thank you!
Repository: allenai/reward-bench
Issue: Add new reward model
Root cause: Unimplemented Feature Gap

We provide two new 2B reward models and a 8B reward model.

- Ray2333/GRM-Gemma-2B-rewardmodel-ft 
- Ray2333/Gemma-2B-rewardmodel-ft
- Ray2333/GRM-Llama3-8B-rewardmodel-ft


## Introduction
This reward model is finetuned from the [Ray2333/GRM-Gemma-2B-sftreg](https://huggingface.co/Ray2333/GRM-Gemma-2B-sftreg) using the [Skywork preference dataset](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1). 

The Skywork preference dataset demonstrates that a small high-quality dataset can lead to powerful reward models, which is promising. By finetuning [Ray2333/GRM-Gemma-2B-sftreg](https://huggingface.co/Ray2333/GRM-Gemma-2B-sftreg) on this dataset, we obtain a **SOTA 2B reward model** that can even surpass gpt4 as a judge. 



## Evaluation
We evaluate GRM-Gemma-2B-rewardmodel-ft on the [reward model benchmark](https://huggingface.co/spaces/allenai/reward-bench), where it achieved SOTA performance among models smaller than 6B.


|       Model               | Average       |  Chat     |     Chat Hard      |     Safety      |     Reasoning     |   
|:-------------------------:|:-------------:|:---------:|:---------:|:--------:|:-----------:|
|**Ray2333/GRM-Gemma-2B-rewardmodel-ft (Ours, 2B)**|  **84.7** | 89.4 | 75.2 | 85.5 | 88.8 |
| openai/gpt-4o-2024-05-13 | 84.6|	96.6	| 70.4	| 86.5	| 84.9 |
| sfairXC/FsfairX-LLaMA3-RM-v0.1 (8B) | 84.4	| 99.4 |	65.1 |	86.8	| 86.4 |
| Nexusflow/Starling-RM-34B (34B)	|	82.6	|96.9	|57.2	|87.7	|88.5|
| **Ray2333/Gemma-2B-rewardmodel-ft (Ours, 2B)** | 80.5 | 77.9 | 74.8 | 85.2 | 84.0 |
|  **Ray2333/GRM-Gemma-2B-sftreg** (2B) | 75.3    |   95.5  |  48.7 |   80.0 | 76.8     |  
|    berkeley-nest/Starling-RM-7B-alpha      (7B)                          |    74.6      |   98      |   43.4   |   88.6  |    74.6    |  
|  **Ray2333/Gemma-2B-rewardmodel-baseline**(2B) | 73.7    |   94.1  |  46.1 |  79.6 |  75.0   |  
|     stabilityai/stablelm-zephyr-3b             (3B)                                 |    73.1      |   86.3   |   60.1   |   70.3  |    75.7     | 


## Run 

**When evaluated using reward bench, please add '--not_quantized' to avoid performance drop.**

CUDA_VISIBLE_DEVICES=0 python scripts/run_rm.py --model=Ray2333/GRM-Gemma-2B-rewardmodel-ft  --chat_template=gemma    --batch_size=${batch_size}  --not_quantized

CUDA_VISIBLE_DEVICES=0 python scripts/run_rm.py --model=Ray2333/Gemma-2B-rewardmodel-ft  --chat_template=gemma    --batch_size=${batch_size}  --not_quantized

CUDA_VISIBLE_DEVICES=0 python scripts/run_rm.py --model=Ray2333/GRM-Llama3-8B-rewardmodel-ft --chat_template=llama-3    --batch_size=${batch_size}  --not_quantized
Repository: allenai/reward-bench
Issue: Add A New Generative Model 
Root cause: Unimplemented Feature Gap

Hi RewardBench Team 👋,

We have updated a 70B version generative model:
- [Skywork/Skywork-Critic-Llama-3.1-70B](https://huggingface.co/Skywork/Skywork-Critic-Llama-3.1-70B)

Our local evaluation metrics for the model is listed as bellow:
```json
{
    "Chat": 0.9692737430167597, 
    "Chat Hard": 0.8837719298245614, 
    "Safety": 0.9324324324324325, 
    "Reasoning": 0.9542546516069188
}
```

Our hardware and environments:
> NVIDIA-SMI 470.199.02
> Driver Version: 470.199.02 
> CUDA Version: 12.3   
> NVIDIA A800-SXM4 80G * 4

### How to Run Evaluation Script
For this generative model, it's okay to evaluate it with the default scripts/run_generative.py script. *Please notice that we need at least 4 gpus to run `run_generative.py` script, and `export VLLM_WORKER_MULTIPROC_METHOD=spawn` is required for vLLM multi-gpu inference.*

```bash
export VLLM_WORKER_MULTIPROC_METHOD=spawn
cd reward-bench
model_name_or_path="Skywork/Skywork-Critic-Llama-3.1-70B"
python scripts/run_generative.py --model $model_name_or_path --trust_remote_code --do_not_save --force_local --num_gpus 4 2>&1 | tee ./evaluation_logs.txt
```

We would like to add this new generative model to the RewardBench LeaderBoard.

Thank you!

@ToSev7en I ran this, thx! It'll be there upon leaderboard restart.
Repository: allenai/reward-bench
Issue: Add A New Generative Model
Root cause: Unimplemented Feature Gap

Hi RewardBench Team 👋,

We have updated a 12B version generative model:

[SF-Foundation/TextEval-OffsetBias-12B](https://huggingface.co/SF-Foundation/TextEval-OffsetBias-12B)
Our local evaluation metrics for the model is listed as bellow:

{
    'Chat': 0.9217877094972067,
    'Chat Hard': 0.868421052631579, 
    'Safety': 0.9238221130221129, 
    'Reasoning': 0.937493179461996
}

How to Run Evaluation Script
For this generative model, it's okay to evaluate it with the default scripts/run_generative.py script. Please notice that we need at least 8 gpus to run run_generative.py script, and export VLLM_WORKER_MULTIPROC_METHOD=spawn is required for vLLM multi-gpu inference.

```
export VLLM_WORKER_MULTIPROC_METHOD=spawn
cd reward-bench
python scripts/run_generative.py --model=SF-Foundation/TextEval-OffsetBias-12B --num_gpus 8
```
We would like to add this new generative model to the RewardBench LeaderBoard.

Thank you!


Reproduced! Will be added when we restart next!
Repository: allenai/reward-bench
Issue: Add A New Generative Model
Root cause: Unimplemented Feature Gap

Hi RewardBench Team 👋,

We have updated a 70B version generative model:

[SF-Foundation/TextEval-70B](https://huggingface.co/SF-Foundation/TextEval-70B)
Our local evaluation metrics for the model is listed as bellow:

{'Chat': 0.946927374301676, 'Chat Hard': 0.9035087719298246, 'Safety': 0.9318428922428922, 'Reasoning': 0.9646321001800622}

How to Run Evaluation Script
For this generative model, it's okay to evaluate it with the default scripts/run_generative.py script. Please notice that we need at least 8 gpus to run run_generative.py script, and export VLLM_WORKER_MULTIPROC_METHOD=spawn is required for vLLM multi-gpu inference.

export VLLM_WORKER_MULTIPROC_METHOD=spawn
cd reward-bench
python scripts/run_generative.py --model=SF-Foundation/TextEval-70B --num_gpus 8
We would like to add this new generative model to the RewardBench LeaderBoard.

Thank you!

Forget to stress the code! Sorry for that.

```
export VLLM_WORKER_MULTIPROC_METHOD=spawn
cd reward-bench
python scripts/run_generative.py --model=SF-Foundation/TextEval-70B --num_gpus 8
Repository: allenai/reward-bench
Issue: Add A New Generative Model
Root cause: Unimplemented Feature Gap

Hi RewardBench Team 👋,

We have updated a 70B version generative model:

[SF-Foundation/TextEval-Llama3.1-70B](SF-Foundation/TextEval-Llama3.1-70B)
Our local evaluation metrics for the model is listed as bellow:

{'Chat': 0.946927374301676, 'Chat Hard': 0.9035087719298246, 'Safety': 0.9318428922428922, 'Reasoning': 0.9646321001800622}

How to Run Evaluation Script
For this generative model, it's okay to evaluate it with the default scripts/run_generative.py script. Please notice that we need at least 8 gpus to run run_generative.py script, and export VLLM_WORKER_MULTIPROC_METHOD=spawn is required for vLLM multi-gpu inference.

```
export VLLM_WORKER_MULTIPROC_METHOD=spawn
cd reward-bench
python scripts/run_generative.py --model=SF-Foundation/TextEval-Llama3.1-70B --num_gpus 8
```

We would like to add this new generative model to the RewardBench LeaderBoard.

Thank you!

@ZhichaoWang970201 will be updated upon restart.
Repository: allenai/reward-bench
Issue: Add a new generative model

We would like to add Con-J-Qwen2-7B by
```
python scripts/run_generative.py --model="ZiyiYe/Con-J-Qwen2-7B"
```
This is our local test results:
{'Chat': 0.9189944134078212, 'Chat Hard': 0.7982456140350878, 'Safety': 0.8797297297297297, 'Reasoning': 0.881871692039068}
Thanks for your help in advance.

Done! Thanks for committing your code directly to the library.
Repository: allenai/reward-bench
Issue: Add a new reward model and make some modifications to reward-bench code
Root cause: Documentation Deficiency

Hi RewardBench Team,

We have updated a 8B reward model (Custom Classifier) [general-preference/GPM-Llama-3.1-8B](https://huggingface.co/general-preference/GPM-Llama-3.1-8B) and a 2b reward model (Custom Classifier) [general-preference/GPM-Gemma-2B](https://huggingface.co/general-preference/GPM-Gemma-2B).

Local evaluation results for our models are listed as bellow:

For [general-preference/GPM-Llama-3.1-8B](https://huggingface.co/general-preference/GPM-Llama-3.1-8B):
{'Chat': 0.9329608938547486, 'Chat Hard': 0.8859649122807017, 'Safety': 0.9055003159003159, 'Reasoning': 0.9597485949691711} 

For [general-preference/GPM-Gemma-2B](https://huggingface.co/general-preference/GPM-Gemma-2B):
{'Chat': 0.7150837988826816, 'Chat Hard': 0.6973684210526315, 'Safety': 0.810949104949105, 'Reasoning': 0.7550369673159819}

We have made some modifications to RewardBench code (v0.1.2), which includes:
1. Incorporate a custom RewardBench pipeline.
2. Create a tailored CustomRewardModel class along with a function to load our model.
3. Implement a custom method for calculating results (scores) for our General Preference model.
4. Introduce additional custom arguments essential for our model.

We have integrated code adapted from RewardBench (v0.1.2) into our repository, [general-preference-model](https://github.com/general-preference/general-preference-model), specifically under `rewardbench_eval` directory.

I would like to inquire about the best practices for incorporating RewardBench with our modifications into the repository. Additionally, we are interested in adding this new reward model to the RewardBench Leaderboard.

Thank you for your time and help!

Best regards,
Grace Zhang

Hey @kirigayahitsugi, there's really similar discussion to this in #198 -- let me know if you need more information.
There is a brief instruction on adding custom pipelines here: https://github.com/allenai/reward-bench/tree/main/rewardbench/models

Thank you for your response. May I modify the pipeline-related components in my code following the instruction you mentioned and submit both those changes and any existing modifications to a new feature branch in a PR?

@kirigayahitsugi - yes that's the normal process!

Thank you for your reply!
Repository: allenai/reward-bench
Issue: DATASET TRACKING
Root cause: Unimplemented Feature Gap

A quick place to list datasets we need to add still.

## Existing Test Sets
* [MT Bench Human Judgements](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments)
* [HHH Alignment](https://huggingface.co/datasets/HuggingFaceH4/hhh_alignment)
* [Auto J](https://github.com/GAIR-NLP/auto-j#pairwise-response-comparison)

Completed:

## New Eval Tools
* Science Data from David Wadden

Completed:
* [HumanEvalPack](https://huggingface.co/datasets/bigcode/humanevalpack)
* Many others earlier in process

(this does not yet include filtering)

This was never really used.
Repository: allenai/reward-bench
Issue: Add new reward models
Root cause: Unimplemented Feature Gap

Currently, no reward models smaller than 7B have ranked in the top 30 on Reward-Bench. We provide new 2B and 3B reward models that are comparable to large-scale reward models.

**Please add the following code into reward-bench/rewardbench/models/__init__.py (other two fine-tuned models do not need this modification)**.

```
"Ray2333/GRM-Gemma2-2B-sftreg": {
        "model_builder": GRewardModel.from_pretrained,
        "pipeline_builder": GRMPipeline,
        "quantized": False,
        "custom_dialogue": False,
        "model_type": "Seq. Classifier",
    },
    "Ray2333/GRM-llama3.2-3B-sftreg": {
        "model_builder": GRewardModel.from_pretrained,
        "pipeline_builder": GRMPipeline,
        "quantized": False,
        "custom_dialogue": False,
        "model_type": "Seq. Classifier",
    },
```

Training Details:
Ray2333/GRM-llama3.2-3B-sftreg is trained on hendrydong/preference_700K.
Ray2333/GRM-Gemma2-2B-sftreg is trained on weqweasdas/preference_dataset_mixture2_and_safe_pku.
Ray2333/GRM-Llama3.2-3B-rewardmodel-ft and Ray2333/GRM-gemma2-2B-rewardmodel-ft are fine-tuned on the **decontaminated dataset** Skywork/Skywork-Reward-Preference-80K-v0.2.

The evaluation commands are:

CUDA_VISIBLE_DEVICES=0 python scripts/run_rm.py --model=Ray2333/GRM-Gemma2-2B-sftreg   --batch_size=8 --not_quantized

CUDA_VISIBLE_DEVICES=0 python scripts/run_rm.py --model=Ray2333/GRM-llama3.2-3B-sftreg   --batch_size=8 --not_quantized

CUDA_VISIBLE_DEVICES=0 python scripts/run_rm.py --model=Ray2333/GRM-gemma2-2B-rewardmodel-ft  --batch_size=8 --not_quantized 

CUDA_VISIBLE_DEVICES=0 python scripts/run_rm.py --model=Ray2333/GRM-llama3.2-3B-rewardmodel-ft  --batch_size=8 --not_quantized 


The local scores are:

|       Model               | Average       |  Chat     |     Chat Hard      |     Safety      |     Reasoning     |   
|:-------------------------:|:-------------:|:---------:|:---------:|:--------:|:-----------:|
|[Ray2333/GRM-Llama3.2-3B-rewardmodel-ft](https://huggingface.co/Ray2333/GRM-Llama3.2-3B-rewardmodel-ft)**(3B)**|90.9|91.6|84.9|92.7|94.6|
| [Ray2333/GRM-gemma2-2B-rewardmodel-ft](https://huggingface.co/Ray2333/GRM-gemma2-2B-rewardmodel-ft) **(2B)**| 88.4 | 93.0 | 77.2 | 92.2 | 91.2 |
|[Ray2333/GRM-llama3.2-3B-sftreg](https://huggingface.co/Ray2333/GRM-llama3.2-3B-sftreg)**(3B)**|85.8|96.4|67.1|88.2|91.6|
|  [Ray2333/GRM-Gemma2-2B-sftreg](https://huggingface.co/Ray2333/GRM-Gemma2-2B-sftreg)**(2B)** | 81.0 |  97.2    |  59.6 | 86.9 |   80.3 |
Repository: allenai/reward-bench
Issue: Add A Series of New Generative Models
Root cause: Unimplemented Feature Gap

Hi RewardBench Team 👋,

We have updated a series of generative models:
[opencompass/CompassJudger-1-32B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-32B-Instruct)
[opencompass/CompassJudger-1-14B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-14B-Instruct)
[opencompass/CompassJudger-1-7B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-7B-Instruct)
[opencompass/CompassJudger-1-1.5B-Instruct](https://huggingface.co/opencompass/CompassJudger-1-1.5B-Instruct)

Our local evaluation metrics for the model is listed as bellow:
For **CompassJudger-1-32B-Instruct**:
{'Chat': 0.9776536312849162, 'Chat Hard': 0.6557017543859649, 'Safety': 0.8613455949455949, 'Reasoning': 0.9221162765318929}
For **CompassJudger-1-14B-Instruct**:
{'Chat': 0.9748603351955307, 'Chat Hard': 0.6228070175438597, 'Safety': 0.8453248157248157, 'Reasoning': 0.9247200169149343}
For **CompassJudger-1-7B-Instruct**:
{'Chat': 0.9776536312849162, 'Chat Hard': 0.6052631578947368, 'Safety': 0.8470778518778519, 'Reasoning': 0.8951922027609537}
For **CompassJudger-1-1.5B-Instruct**:
{'Chat': 0.9636871508379888, 'Chat Hard': 0.49451754385964913, 'Safety': 0.7813390663390664, 'Reasoning': 0.6959131336279806}

To Run Evaluation Script
For these generative models, it's okay to evaluate them with the default scripts/run_generative.py script. Just as below code:
```
export VLLM_WORKER_MULTIPROC_METHOD=spawn
cd reward-bench
python scripts/run_generative.py --model=opencompass/CompassJudger-1-32B-Instruct --num_gpus 4
```
We would like to add these series of new generative models to the RewardBench LeaderBoard.

Thank you!

Running now. Will be visible upon next restart.
Repository: allenai/reward-bench
Issue: Add new model INF-ORM-Llama3.1-70B
Root cause: Unimplemented Feature Gap

We trained a new model called "INF-ORM-Llama3.1-70B".
https://huggingface.co/infly/INF-ORM-Llama3.1-70B

Hi, RewardBench Team! 
**Supplementary information**
I have submitted a PR to add some code for evaluating our model. We trained it on the dataset [INF-ORM-Preference-Magnitude-80K](https://huggingface.co/datasets/infly/INF-ORM-Preference-Magnitude-80K), which is derived from the **decontaminated dataset** [Skywork/Skywork-Reward-Perference-80k-v0.2](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.2).
**Evaluation commands**
We evaluated our model with 8*A100 GPUs.
 python scripts/run_rm.py \
 --model=infly/INF-ORM-Llama3.1-70B \
 --tokenizer=infly/INF-ORM-Llama3.1-70B \
 --batch_size=16 \
 --torch_dtype=bfloat16 \
 --attn_implementation=flash_attention_2 \
 --not_quantized \
**Local scores**
| Model                                        | Model Type        | Score | Chat  | Chat Hard | Safety | Reasoning |
| -------------------------------------------- | ----------------- | :---: | :---: | :-------: | :----: | :-------: |
| **infly/INF-ORM-Llama3.1-70B**  | Custom Classifier   | 95.2  | 96.9  |   91.0    |  93.8  |   99.1    |

Hello, I'm a research assist in Tsinghua Shenzhen International Graduated School, we evaluated the reward model using the same instructions on 8*A100 GPUs. However, the results are not the same as you say, and even the two adjacent experiments reflect large differences.
Here are the results we got from one of the runs:
{'Chat': 0.6759776536312849, 'Chat Hard': 0.8771929824561403, 'Safety': 0.9162162162162162, 'Reasoning': 0.9216456594096143}  
I'm wondering the reason for this phenomenon, is it because of temperature or random seeds?
Looking forward to your reply, thanks!


Hi, it seems that the model weights have not been initialized correctly. Make sure that all the model weights are loaded correctly. And could you provide more information about it?
Repository: allenai/reward-bench
Issue: Basic questions with using the repo for Llama and Mistral models
Root cause: Documentation Deficiency

Hi team,

Great work!! Really appreciate bringing so many details together. As a beginner, I've some basic questions, so appreciate your patience.

If I understand it correctly, a LM can be evaluated 

- Either as a classifier depending on how they were trained - if a normal reward model is used, then we will be using `run_rm.py`, and if DPO is used, we will be using `run_dpo.py` 
- Or generate the responses from this LM, use another LM (GPT4 or so as a judge) and evaluate the generative responses - `run_generative.py`

Now, if my understanding is correct, Llama models (eg: `meta-llama/Llama-2-7b-chat-hf`) is trained using a reward model and Mistral models (eg: `mistralai/Mistral-7B-Instruct-v0.2`) is trained using DPO. 

**Q:** So my first question is, `run_rm.py` can be used for Llama models and `run_dpo.py` can be used for Mistral models. 

**Q:** If instead of evaluating in classifier fashion, I plan to evaluate using the generative approach which will truly tell the capability of LM, I believe the below commands should work, without providing any chat templates!

```
python scripts/run_generative.py --model=meta-llama/Llama-2-7b-chat-hf --force_local
python scripts/run_generative.py --model=mistralai/Mistral-7B-Instruct-v0.2 --force_local
```
I'm aware that the main results in the paper (except for Table 8) are of classifier based reward models; and as such the final results when using `run_generative.py` will be lower as generative based approach is more challenging.

The big distinction is on the type of LM it is. You can think about this with HuggingFace Transformers abstractions. Most reward models are trained with `AutoModelForSequenceClassification` which adds a value head to output one logit -- these are run with `run_rm` (along with similar models with slightly different architecture). A standard generative model is `AutoModelForCausalLM` which is run with `run_generative`. 

`run_dpo` is largely deprecated, but it is designed to run a DPO trained model as an implicit RM. These models can also be run with `run_generative`, but the usages as a RM is different (generative == llm as a judge). 

Hope that helps @NamburiSrinath