Repository: EvolvingLMMs-Lab/lmms-eval
Issue: 03-25 13:08:56 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
Root cause: External Dependency Breakage

03-25 13:09:00 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:279] ERROR Error Invalid key: 3139 is out of bounds for size 2801 in getting visuals, task_type is multidocvqa_test)
Traceback (most recent call last):
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/__main__.py", line 199, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/__main__.py", line 283, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
    return fn(*args, **kwargs)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 128, in simple_evaluate
    results = evaluate(
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
    return fn(*args, **kwargs)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 290, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 280, in generate_until
    raise e
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in generate_until
    visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in <listcomp>
    visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2810, in __getitem__
    return self._getitem(key)
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2794, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices)
  File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 3139 is out of bounds for size 2801
03-25 13:09:03 [lmms-eval/lmms_eval/__main__.py:213] ERROR Error during evaluation: Invalid key: 3139 is out of bounds for size 2801
Traceback (most recent call last):
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/__main__.py", line 199, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/__main__.py", line 283, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
    return fn(*args, **kwargs)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 128, in simple_evaluate
    results = evaluate(
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
    return fn(*args, **kwargs)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 290, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 280, in generate_until
    raise e
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in generate_until
    visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
  File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in <listcomp>
    visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2810, in __getitem__
    return self._getitem(key)
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2794, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices)
  File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 3139 is out of bounds for size 2801




@ChunyuanLI @Luodian @FengLi-ust @jzhang38 @JvThunder   how to resolve it ?

Please stick the transformers version to 4.37.2

We would have later PR to make it compatiable with transformers > 4.37.2