Repository: EvolvingLMMs-Lab/lmms-eval
Issue: issues with batch_size (Index error)
Root cause: Validation Gap

I really appreciate your good work.
I'm reproducing refcoco iou test on qwen2-vl. Below is my script.
```
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5
accelerate launch --num_processes=6 --main_process_port 12345 -m lmms_eval --model qwen2_vl   --model_args pretrained="/data/student007/hf_downloads/modules/Qwen2-VL-7B-Instruct/
"   --tasks refcoco  --batch_size 3 --log_samples --log_samples_suffix debug --output_path ./logs/qwen2_vl_7b 
```
When batch_size is 1, it works well. But when batch_size is larger than 1, I will encounter error message like:
```
Model Responding:   0%|▏                                                                              | 13/5973 [00:11<1:15:18,  1.32it/s]Traceback (most recent call last):
  File "/data/student007/projects/lmms-eval/lmms_eval/__main__.py", line 330, in cli_evaluate
    results, samples = cli_evaluate_single(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/student007/projects/lmms-eval/lmms_eval/__main__.py", line 471, in cli_evaluate_single
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/student007/projects/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/student007/projects/lmms-eval/lmms_eval/evaluator.py", line 248, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/data/student007/projects/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/student007/projects/lmms-eval/lmms_eval/evaluator.py", line 458, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/student007/projects/lmms-eval/lmms_eval/models/qwen2_vl.py", line 172, in generate_until
    visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/student007/projects/lmms-eval/lmms_eval/models/qwen2_vl.py", line 172, in <listcomp>
    visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/data/student007/miniconda3/envs/qwen2_5_eval/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2800, in __getitem__
    return self._getitem(key)
           ^^^^^^^^^^^^^^^^^^
  File "/data/student007/miniconda3/envs/qwen2_5_eval/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2784, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/student007/miniconda3/envs/qwen2_5_eval/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/data/student007/miniconda3/envs/qwen2_5_eval/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 14650 is out of bounds for size 5657
```
And --batch_size "auto" or --batch_size auto will also raise some error message.
I don't it's because I make a mistake or it's a bug.



same question