Repository: allenai/reward-bench
Issue: text_chosen is not shown
Root cause: Validation Gap

python scripts/run_generative_v2.py --num_gpus=8 --model=Qwen/Qwen3-32B --dataset=allenai/reward-bench-2


INFO 06-12 02:42:39 [core.py:167] init engine (profile, create kv cache, warmup model) took 925.38 seconds
2025-06-12 02:42:46 - INFO - __main__ - *** Load dataset ***
2025-06-12 02:42:47 - INFO - __main__ - *** Preparing dataset with custom formatting ***
2025-06-12 02:42:48 - INFO - __main__ - Dataset columns: ['id', 'prompt', 'chosen', 'rejected', 'num_correct', 'num_incorrect', 'total_completions', 'models', 'subset', 'additional_metadata', 'texts_chosen', 'texts_rejected']
Map:   0%|                                                                                                                       | 0/1763 [00:00<?, ? examples/s]
Traceback (most recent call last):
  File "/mnt/vast/home/jimmy/evaluation/reward-bench/scripts/run_generative_v2.py", line 545, in <module>
    main()
  File "/mnt/vast/home/jimmy/evaluation/reward-bench/scripts/run_generative_v2.py", line 431, in main
    dataset_prompts = dataset.map(format_judgements, fn_kwargs={"optional_chat_template": chat_template})
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/.venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/.venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3079, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/.venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3501, in _map_single
    for i, example in iter_outputs(shard_iterable):
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/.venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3475, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/.venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3398, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/reward-bench/scripts/run_generative_v2.py", line 376, in format_judgements
    mult_turn = True if len(batch["text_chosen"]) > 2 else False
                            ~~~~~^^^^^^^^^^^^^^^
  File "/mnt/vast/home/jimmy/evaluation/.venv/lib/python3.12/site-packages/datasets/formatting/formatting.py", line 278, in __getitem__
    value = self.data[key]
            ~~~~~~~~~^^^^^
KeyError: 'text_chosen'

Is this the correct dataset to be used for the rewardbench v2 leaderboard?

Yes @Zhichao-Jimmy970201  -- we're fixing some of this with #249. Will be merged ASAP

Hi @Zhichao-Jimmy970201 — this should be fixed!