zhiminy/EvalEng huggingface dataset:
Repository: allenai/reward-bench
Issue: Generative RM
Root cause: Unimplemented Feature Gap

To use models like GPT4 and others as a baseline, we need a script that generates a response to which is better.
I'm not sure if we want to include this yet.

An example model is [Auto J](https://github.com/GAIR-NLP/auto-j?tab=readme-ov-file#usage).

Even with temperature = 0, there are lots of ways for this to seem unnecessary and non-deterministic (unless trained with DPO).
Repository: allenai/reward-bench
Issue: Fix score saving PairRM and SteamSHP
Root cause: Algorithmic Error

The logic for saving per-prompt scores is breaking these two models, see the following beaker logs. 
https://beaker.org/ex/01HQ6VG7H3XPYRVP3S76ZXB126
https://beaker.org/ex/01HQ6VG7GMRN9TNWNZH2WTMKG0

A simple fix should be to change the following line in `run_rm.py`

```
                scores_chosen.extend(None * len(results_sub))
                scores_rejected.extend(None * len(results_sub))
```
to
```
                scores_chosen.extend([0] * len(results_sub))
                scores_rejected.extend([0] * len(results_sub))
```

Has been fixed in #31 (I think)
Repository: allenai/reward-bench
Issue: Check Qwen model
Root cause: Algorithmic Error

Results are quite odd, initial support added in #47 following [tokenizer documentation](https://github.com/QwenLM/Qwen/blob/main/tokenization_note.md) and [FAQ](https://github.com/QwenLM/Qwen/blob/044c06b244273a3af1abe0ec93c6edba14e481cf/FAQ.md?plain=1#L82) of the model.

Results are worse than expected.

Closing as doing more eval it seems correct. SOTA on chat hard, just something weird on the easier ones.
Repository: allenai/reward-bench
Issue: Check beaver cost model
Root cause: Algorithmic Error

Quoting an author (I think):
> Great work, this is an long due effort in this field. Though it's a bit unexpected to see beaver-cost model performed poorly on safety-related dataset. Have you checked if you have got the signs worked out? Because in our setting negative reward means safer and should be chosen.


Thanks for the quick fix!! (cc: @XuehaiPan)
Repository: allenai/reward-bench
Issue: Add Nvidia RMs (and Nemo compatibility)
Root cause: Unimplemented Feature Gap

See below!
https://huggingface.co/models?library=nemo&search=RM

This involves converting to HF format or adding nemo compatibility, if anyone has time!

Closing because stale.

Hello @natolambert 

Is there any update on this? Or do you know if anyone has used Nvidia nemo RMs in Reward-Bench or M-Reward-Bench for compatibility?



@Riashat no update. Thankfully Nvidia has been releasing more -hf checkpoints, so less of an issue now.
Repository: allenai/reward-bench
Issue: Add new model Mistral-7B-instruct-Unified-Feedback
Root cause: Unimplemented Feature Gap

Thank you for your great work! Could you add my reward model on the leader board? I have tested it locally.

[Ray2333/reward-model-Mistral-7B-instruct-Unified-Feedback](https://huggingface.co/Ray2333/reward-model-Mistral-7B-instruct-Unified-Feedback)
Repository: allenai/reward-bench
Issue: Add a new mistral RM model
Root cause: Unimplemented Feature Gap

Thank you for your work! Can you please test my RM `hendrydong/Mistral-RM-for-RAFT-GSHF-v0` in the leaderboard?

My local results are as below:
```
{"model": "hendrydong/Mistral-RM-for-RAFT-GSHF-v0", "model_type": "Seq. Classifier", "chat_template": "tokenizer", "alpacaeval-easy": 0.99, "alpacaeval-hard": 1.0, "alpacaeval-length": 0.9473684210526315, "donotanswer": 0.6470588235294118, "hep-cpp": 0.9390243902439024, "hep-go": 0.9573170731707317, "hep-java": 0.9695121951219512, "hep-js": 0.9390243902439024, "hep-python": 0.9451219512195121, "hep-rust": 0.9329268292682927, "llmbar-adver-GPTInst": 0.3804347826086957, "llmbar-adver-GPTOut": 0.5957446808510638, "llmbar-adver-manual": 0.34782608695652173, "llmbar-adver-neighbor": 0.4701492537313433, "llmbar-natural": 0.9, "math-prm": 0.5503355704697986, "mt-bench-easy": 1.0, "mt-bench-hard": 0.7837837837837838, "mt-bench-med": 0.975, "refusals-dangerous": 0.75, "refusals-offensive": 0.96, "xstest-should-refuse": 0.9805194805194806, "xstest-should-respond": 0.888}
```



Trying it @hendrydong -- next time can you add a little more documentation to the model card / issue to make sure I'm using it correctly? Thx.
Repository: allenai/reward-bench
Issue: Add new model weqweasdas/RM-Mistral-7B
Root cause: Unimplemented Feature Gap

Hi! Congratulations on your amazing work!

I am wondering whether it is convenient for you to add this reward model in your leaderboard: 

https://huggingface.co/weqweasdas/RM-Mistral-7B

many thanks in advance!
Repository: allenai/reward-bench
Issue: stanfordnlp/SteamSHP-flan-t5 performance on SHP and HH-RLHF Helpful
Root cause: Validation Gap

Hi, thanks for this great work, its really interessting and helpful!

I was a bit surprised by the stanfordnlp/SteamSHP-flan-t5-xl and stanfordnlp/SteamSHP-flan-t5-large performance on the SHP dataset in Table 12, because their self reported accuracy is [0.7278](https://huggingface.co/stanfordnlp/SteamSHP-flan-t5-xl#:~:text=ALL%20(unweighted),0.7278) and [0.7203](https://huggingface.co/stanfordnlp/SteamSHP-flan-t5-large#:~:text=ALL%20(unweighted),0.7203) respectively. Do you know the reason for this difference?

(AFAIK, their reported average also includes the performance on HH-RLHF helpful-base, but I dont think that should drag the performance down that much?)

Vice versa, the HH-RLHF helpful scores in Table 12 are much lower than the reported ones on huggingface (0.731 vs 0.633 and 0.731 vs 0.629).

<img width="935" alt="Screenshot 2024-03-22 at 15 26 07" src="https://github.com/allenai/reward-bench/assets/11020443/bc2e2352-270b-4064-be92-819b9ffd4742">





I'll look further @timbmg. Some specific things, some open questions here.
1. Our SHP test set is a smaller curated subset (as it would be huge otherwise). From the [prior pref sets dataset card](https://huggingface.co/datasets/allenai/preference-test-sets). In short, we make the test set less noisy (I'm happy to see the numbers are higher tbh):
> [Stanford Human Preferences (SHP)](https://huggingface.co/datasets/stanfordnlp/SHP), with a subset created by taking 1 sample per prompt with a score ratio above 1.5 and a total number of Reddit votes above 10.
2. I feel like Anthropic HH can catch a lot of people on chat templates. We should check their implementation for the dataset.
3. Given the SHP model is a little weird to run, there could be bugs. If you have time to check our implementation here it would be great. It is mostly copied from their code tbf.
https://github.com/allenai/reward-bench/blob/main/rewardbench/models/shp.py
Repository: allenai/reward-bench
Issue: Clean up / enhance DPO code
Root cause: Unimplemented Feature Gap

1. Make it so you can run inference over individual text prompts (rather than chosen + rejected)
2. Clean up nograd/detach (see https://twitter.com/shxf0072/status/1771220126655811610), but should be pretty obvious
3. Add per-model log prob computation. 

Closing because stale. Feel free to reopen.
Repository: allenai/reward-bench
Issue: adding Archangel models (dpo, kto, sft+dpo, sft+kto)
Root cause: Unimplemented Feature Gap

The Archangel suite of models contain DPO, SFT+DPO, KTO, SFT+KTO models which can also be used as reward models: https://huggingface.co/collections/ContextualAI/archangel-65bd45029fa020161b052430

For each method, there are seven models available: pythia-{1.4, 2.8, 6.9, 12.0}B and llama-{7, 13, 30}B, all of which have been aligned under nearly identical settings on {Anthropic HH, Open Assistant, SHP 1.0} data.

The implied reward for both DPO- and KTO-aligned models is $\beta \log \frac{\pi_\theta(y|x)}{\pi_\text{ref}(y|x)}$, where $\pi_\text{ref}$ is the reference model

The reference model for each set of models in Archangel is as follows:
- for the SFT+DPO model `ContextualAI/archangel_sft-dpo_{model}`, the reference is `ContextualAI/archangel_sft_{model}`
- for the SFT+KTO model `ContextualAI/archangel_sft-kto_{model}`, the reference is `ContextualAI/archangel_sft_{model}`
- for the DPO model w/o SFT `ContextualAI/archangel_dpo_llama7b`, the reference is `huggyllama/llama-7b`, which can be found in the `_name_or_path` field in `config.json`
- for the KTO model w/o SFT `ContextualAI/archangel_kto_llama7b`, the reference is `huggyllama/llama-7b`, which can be found in the `_name_or_path` field in `config.json`
Repository: allenai/reward-bench
Issue: Saving bug (non breaking)
Root cause: Algorithmic Error

We don't use our own sub_path correctly in saving results. It works, but is confusing.
See:
1. https://github.com/allenai/reward-bench/blob/main/scripts/run_rm.py#L293
2. https://github.com/allenai/reward-bench/blob/main/rewardbench/utils.py#L50

Tbh i'm surprised the code still works as expected (lol)
Repository: allenai/reward-bench
Issue: Check EOS token on FastChat models
Root cause: Configuration Error

TLDR:
* HF tokenizer files add EOS token at the end of instructions https://huggingface.co/HuggingFaceH4/zephyr-7b-beta/blob/b70e0c9a2d9e14bd1e812d3c398e5f313e93b473/tokenizer_config.json#L34
* DPO code manually adds EOS token https://github.com/huggingface/trl/blob/6c2f829bb7408660b0e581cde56fbff0980b9d7b/trl/trainer/dpo_trainer.py#L668

Seems like FastChat models are not getting an EOS token. Could effect the non-DPO models that are using FastChat chat templates minorly

Solution for the relevant models (will need more filtering code)
```
    ############################
    # Tokenization settings & dataset preparation
    ############################
    # set pad token to eos token if not set
    if reward_pipe.tokenizer.pad_token_id is None:
        reward_pipe.model.config.pad_token_id = reward_pipe.tokenizer.unk_token_id
        reward_pipe.tokenizer.pad_token_id = reward_pipe.tokenizer.unk_token_id
    # make tokenizer add eos
    reward_pipe.tokenizer.add_eos_token = True
```
Repository: allenai/reward-bench
Issue: multi gpu inference with run_rm.py
Root cause: Unimplemented Feature Gap

Hello Nathan,

Thank you for this valuable resource! I strongly think that we needed more standardized benchmarks to evaluate reward/evaluator models.

I think submit_eval_jobs.py (using AI2's beaker) supports multi gpu inference but run_rm.py doesn't at the moment.
I was wondering if this intended (correct me if I'm wrong)!

Best,
Seungone

Hey @SeungoneKim -- we just haven't needed it yet (the biggest classifiers are 34B). Happy to add it.

`run_dpo.py` works nicely with 2,4,6,8 GPUs. That's what it's included.
Lmk if you want to open a PR :) 

Thanks for your response @natolambert!

I was trying to test generative reward modeling (with GPT-4, Prometheus, Auto-J) and it seems like `run_dpo.py` has a slightly different functionality than what I need.

Considering that generative RMs require generating a CoT-ish feedback before their scoring decision, I think it would be best to integrate vllm and add an additional `run_generative_rm.py` code. Users could add on additional generative rms by implementing the code for parsing the output(reward).

If this makes sense to you, I'll leave a pull request of this and try to maintain the style of the code as similar to `run_rm.py`!

@SeungoneKim generative RM's (via API) are being added in #86, but adding the full generation thing is another can of worms. I agree with your path, I just worry a bit about complexity. It's prolly worth having though.

The API implementation should be closer to what you want to build off of.

Here are preliminary results
```
Claude results:
Haiku {‘Chat’: 0.9273743016759777, ‘Chat Hard’: 0.5197368421052632, ‘Safety’: 0.8210275184275184, ‘Reasoning’: 0.7060194658154636}
Sonnet {‘Chat’: 0.9343575418994413, ‘Chat Hard’: 0.5657894736842105, ‘Safety’: 0.8367826605826606, ‘Reasoning’: 0.6907005374583948}
Opus {‘Chat’: 0.946927374301676, ‘Chat Hard’: 0.6030701754385965, ‘Safety’: 0.8905447525447526, ‘Reasoning’: 0.7868223795492989}
(reminder) OpenAI results:
GPT 3.5 {‘Chat’: 0.9217877094972067, ‘Chat Hard’: 0.4451754385964912, ‘Safety’: 0.6229577395577396, ‘Reasoning’: 0.5912315163420091}
GPT 4.Turbo {‘Chat’: 0.952513966480447, ‘Chat Hard’: 0.743421052631579, ‘Safety’: 0.8719219375219376, ‘Reasoning’: 0.8692366453865881}
```
Repository: beir-cellar/beir
Issue: DPR [KILT] checkpoint release plan and which data is used? 

Hello, thanks for the great work and open-sourced code! I have questions about DPR [KILT] model which is recorded on[ BEIR retrieval leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=0).

First, what specific KILT data is used for this model? Is it the same setting as [ Multi-task Retrieval for Knowledge-Intensive Tasks](https://arxiv.org/abs/2101.00117) (8 knowledge-intensive IR tasks)? 
<img width="409" alt="image" src="https://user-images.githubusercontent.com/35963221/179469444-a3fe86b9-37ea-4925-a9d2-97c9c585e4a4.png">

Second, do you have a plan to release the checkpoint for this model?

Thanks!
Repository: beir-cellar/beir
Issue: Inconsistency in Arguana
Root cause: Validation Gap

First let me thank you for the huge work putting this benchmark together.

While downloading and processing the dataset I came accross something weird in the Arguana Dataset

The id `test-free-speech-debate-yfsdfkhbwu-con03b` is considered a relevant passage in the qrels/test.tsv file.
But this id is not present in the `corpus.jsonl` file. 

In the pytrec eval tool used, the tool checks whether the query id is present and if not log something to tell us but it's not the case for passage id.
Thus I think this qrel line will be valid but will never be satisfied during evaluation since the passage id is not in the corpus. 
Is that a normal behavior ? or should it be filtered in the beir original dataset ?

Thanks,
Remi
Repository: beir-cellar/beir
Issue: custom metric implementation depend on length of qrels
Root cause: Documentation Deficiency

Hi @thakur-nandan, Great work! Keep it up! 🍻
Just noticed one thing in the code for the custom metric ([here](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/retrieval/custom_metrics.py#L28)), the sum of MRR@k scores is divided by len(qrels). However, this should actually be divided by the number of queries for which MRR is computed (in your code this can be len(results)). 
Please let me know if I'm wrong. If not, I am happy to create a PR :)

Best,
Iknoor Singh

Hi @iknoorjobs, 

From my understanding, the evaluation should be based on the length of ``qrels``. 

For example, for a few queries in lexical search with Elasticsearch, each query is unable to return any top-k documents. In such cases, the ``results`` dictionary will not include this query. Here we must make sure the retriever is penalized by assigning MRR@k score equal to 0 for these queries present in qrels. 

We always take ``qrels`` to be the ground truth and do evaluations based on it.

Kind Regards,
Nandan Thakur



Hi @thakur-nandan,
Thanks for your response. Yes, that makes sense. In this case, I'm using a custom dataset for evaluation and my `qrels` are combined for the test and train set queries, that's probably what's causing the wrong results.
Regards,
Iknoor
Repository: beir-cellar/beir
Issue: Does BEIR support tuning config k and b of BM25 ?
Root cause: Documentation Deficiency

Hi @thakur-nandan, thanks for the amazing work! I would like to know if BEIR supports tuning config k and b of BM25?
In the paper, anserini BM25 was implemented with default Lucene parameters (k=0.9 and b=0.4). I want to run [evaluate_bm25.py](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py) or  [evaluate_anserini_bm25.py](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_anserini_bm25.py) of beir with different k and b. But I can't find where to tune them.

Thanks.

You can find Anserini regressions for BEIR here: https://github.com/castorini/anserini#regressions-for-beir-v100
You can reuse the tuning script implemented here: https://github.com/castorini/anserini/blob/master/docs/experiments-msmarco-passage.md#bm25-tuning

> You can find Anserini regressions for BEIR here: https://github.com/castorini/anserini#regressions-for-beir-v100 You can reuse the tuning script implemented here: https://github.com/castorini/anserini/blob/master/docs/experiments-msmarco-passage.md#bm25-tuning

Does BEIR support originally setting the k and b for BM25, or should I import the BM25 from Anserini to set these parameters? I find the scripts in the two links are hard to follow. Are there any other relevant tutorials or documentation?

Thanks
Repository: beir-cellar/beir
Issue: Custom metrics handle document score ties differently from pytrec_eval
Root cause: Algorithmic Error

I ran an experiment and there were some inconsistencies in the metrics calculated. In the dataset used there exists only one relevant document per query. In this case, MAP is precisely the same as MRR. To my surprise, I noticed a difference between the MAP and the MRR. Digging into the results, I saw that the difference comes when there are some documents score ties. After looking into the code of pytrec_eval and reading [The Impact of Score Ties on Repeatability
in Document Ranking](https://cs.uwaterloo.ca/~jimmylin/publications/Lin_Yang_SIGIR2019.pdf). I learned that pytrec_eval breaks down ties by sorting the _docids_ in a descending manner. I have fixed the code for my experiment so the two metric calculations match. 

I can create a pull request if needed.

Thanks, 

Olivier
Repository: beir-cellar/beir
Issue: NQ training qrels missing
Root cause: Documentation Deficiency

Hey, I was looking into the training sets contained in BEIR, and it seems that the NQ one is missing. The the one provided in the current link only has the test set (https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq.zip). Could you help me with this?

Hi @cadurosar, 

The `nq` folder provided only contains the test set. For nq training qrels you can have a look into [https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq-train.zip](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq-train.zip).

The reason for this is because the corpus used for both training and test is different. Training set contains passage duplicates which are suitable for training, however not for validation.

Thanks @thakur-nandan!
Repository: beir-cellar/beir
Issue: DenseRetrievalExactSearch:  Out of memory
Root cause: Resource Mishandling

I found BEIR consumes huge amount of memory when evaluating with large datasets such as HotpotQA and MSMARCO.

[L73-L78](https://github.com/beir-cellar/beir/blob/1a1e6aba14b6a2ba1f261a003e7c1ea46fd61564/beir/retrieval/search/dense/exact_search.py#L73) could be optimized since `self.results` will keep accumulating new scores, whereas only top K items will be used for computing metrics. 

Consider changing to the code below:

```
for query_itr in range(len(query_embeddings)):
    query_id = query_ids[query_itr]                  
    for sub_corpus_id, score in zip(cos_scores_top_k_idx[query_itr], cos_scores_top_k_values[query_itr]):
        corpus_id = corpus_ids[corpus_start_idx+sub_corpus_id]
        if corpus_id != query_id:
            self.results[query_id][corpus_id] = score
    _results = {}
    for cid, score in sorted(self.results[query_id].items(), key=lambda k: k[1], reverse=True)[:top_k]:
         _results[cid] = score
    self.results[query_id] = _results
```


Hi @memray, thank you for sharing this, it is really useful to address the memory issue when working with large datasets.
Repository: beir-cellar/beir
Issue: webis-touche2020 qrels
Root cause: Configuration Error

Thanks for providing this resource!

I noticed that the hash of `webis-touche2020.zip` recently changed, so I dug a bit into the differences. It looks like the only changes were changes to `webis-touche2020/qrels/test.tsv`:

```
$ wc -l {old,new}/webis-touche2020/qrels/test.tsv
  2963 old/webis-touche2020/qrels/test.tsv
  2299 new/webis-touche2020/qrels/test.tsv

$ head {old,new}/webis-touche2020/qrels/test.tsv -n5
==> old/webis-touche2020/qrels/test.tsv <==
query-id	corpus-id	score
1	197beaca-2019-04-18T11:28:59Z-00001-000	4
1	1a76ed9f-2019-04-18T16:07:27Z-00001-000	5
1	1a76ed9f-2019-04-18T16:07:27Z-00002-000	3
1	1a76ed9f-2019-04-18T16:07:27Z-00005-000	4

==> new/webis-touche2020/qrels/test.tsv <==
query-id	corpus-id	score
1	S197beaca-A971412e6	0
1	S1b03f390-A22aff8a0	0
1	S1b03f390-Aa73ba80f	1
1	S1b03f390-Ab387b162	0
```

I'm not super familiar with the task, but based on [this page](https://webis.de/events/touche-20/shared-task-1.html#results) and the linked qrels, it looks like the old version corresponds to version 1 of the args.me corpus, and the new version corresponds with version 2. However, the `corpus.jsonl` remains unchanged, so the qrels `corpus-id` field in the qrels no longer corresponds with document `_id` fields in the corpus.

Can you please clarify these discrepancies?

Hi @seanmacavaney,

Thanks for bringing this to notice. The original task authors recently emailed me a new version of the qrels, which I added however I failed to notice it corresponded to version 1 of the corpus. The older version which I hosted all corresponds to version 2 of the corpus. 

For now, I will host back the old dataset zip and speak with the task authors about whether the qrels can be changed for version 2 of the corpus.

Kind Regards,
Nandan

Great- thanks Nandan!

Already changed back to the old version, you can find the dataset here - 
https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/webis-touche2020.zip

Kind Regards,
Nandan

Thanks! My 2cents: If you're planning to eventually go with the new qrels, I'd suggest adding them as a separate benchmark (`webis-touche2020-v2`?), rather than doing a wholesale replacement. Although the [metrics look similar for the different versions of the qrels](https://webis.de/events/touche-20/shared-task-1.html#results), they are not really directly comparable.

Yes, makes sense. Thanks for the suggestion.
This nomenclature I was also thinking to adopt => ``webis-touche2020-v2``
Repository: beir-cellar/beir
Issue: Different version of marco training set
Root cause: Documentation Deficiency

Hi @thakur-nandan, thanks for the great work! 

I am trying to reproduce some of the dense retrievers' results on the  BEIR [leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=0), and I found two versions (v2 and v3) of the processed MS MARCO dataset [here](https://github.com/beir-cellar/beir/tree/main/examples/retrieval/training). I wonder which is the one used to train the dense retrievers? Thanks! 

Following the instructions, I have constructed both v2 and v3 datasets. Both of them have 498970 unique queries and 9144553 (query, positive_passage, negative_passage) triples. Seems they are the same dataset with different data structures. I will close this issue.
Repository: beir-cellar/beir
Issue: the number of queries is not as reported for some datasets?

Hello,

In some datasets, for example FIQA, if I download the datasets from https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip
the queries.jsonl contains 6648 queries, while in the paper or the readme of github, it is reported 648,
could you tell me why they are different？which one is correct

Thanks

Hi @lmh0921, 

The `queries.jsonl` file contains all train, development and test queries together in the single file. Refer to the `qrels/test.tsv` for the 648 queries (unique query ids) present in the test split.

Kind Regards,
Nandan Thakur

ok, got the point,
thanks @thakur-nandan !