Repository: EvolvingLMMs-Lab/lmms-eval
Issue: DocVQA Eval: IndexError: Invalid key: 5205 is out of bounds for size 5188
Root cause: Validation Gap

Facing the error below while evaluating on the docVQA dataset.


```
Traceback (most recent call last):
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/__main__.py", line 206, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/__main__.py", line 301, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/utils.py", line 453, in _wrapper
    return fn(*args, **kwargs)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/evaluator.py", line 135, in simple_evaluate
    results = evaluate(
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/utils.py", line 453, in _wrapper
    return fn(*args, **kwargs)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/evaluator.py", line 297, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/models/llava.py", line 303, in generate_until
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/lmms_eval/models/llava.py", line 303, in <listcomp>
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2800, in __getitem__
    return self._getitem(key)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2784, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/scratch/home/gos2pi/miniconda3/envs/llava/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 5205 is out of bounds for size 5188
```

+1, while using qwen2-vl 

Hi, have you figured out what is the bug?

I have also meet this problem when I test qwen2-vl-7B-instruct on textVQA with batch_size != 1.

oh, I find this [issue](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/32).
I think this info should be highlight.

> oh, I find this [issue](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/32). I think this info should be highlight.

Hi!

I encountered the same issue when evaluating on the TextVQA and MMBench datasets. After reviewing the link you provided, it seems that the issue remains unresolved. Am I correct in understanding that this issue cannot be resolved at the moment?