Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Question about run ScienceQA on llava-1.5-hf
Root cause: Validation Gap

Running following code 
```
accelerate launch --num_processes=4 --main_process_port 12325 -m lmms_eval \
        --model llava_hf \
        --model_args pretrained="/hdd0/model/llava-1.5-7b-hf" \
        --tasks scienceqa \
        --batch_size 1 \
        --output_path "/hdd0/lmms-eval/evaluation_results/llava_v1.5" \
        --log_samples
```
and output
```
Model Responding:   0%|                                                                                                | 0/1061 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/hdd0/bmy/lmms-eval/lmms_eval/__main__.py", line 347, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/hdd0/bmy/lmms-eval/lmms_eval/__main__.py", line 474, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/hdd0/bmy/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "/hdd0/bmy/lmms-eval/lmms_eval/evaluator.py", line 269, in simple_evaluate
    results = evaluate(
  File "/hdd0/bmy/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "/hdd0/bmy/lmms-eval/lmms_eval/evaluator.py", line 502, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/hdd0/bmy/lmms-eval/lmms_eval/models/chat/llava_hf.py", line 72, in generate_until
    inputs = self._image_processor(images=visuals, videos=videos, text=text, return_tensors="pt").to(self._device, self.model.dtype)
  File "/ssd0/bmy/conda_envs/lmmeval/lib/python3.10/site-packages/transformers/models/llava/processing_llava.py", line 150, in __call__
    image_inputs = self.image_processor(images, **output_kwargs["images_kwargs"])
  File "/ssd0/bmy/conda_envs/lmmeval/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 44, in __call__
    return self.preprocess(images, **kwargs)
  File "/ssd0/bmy/conda_envs/lmmeval/lib/python3.10/site-packages/transformers/models/clip/image_processing_clip.py", line 314, in preprocess
    if do_rescale and is_scaled_image(images[0]):
IndexError: list index out of range
```
But when I test Scienceqa using Qwen2.5 using following code, it works, can you fix the problem about llava?
```
accelerate launch --num_processes=4  --main_process_port=21818 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained="/ssd0/Qwen2.5-VL-3B-Instruct",max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False \
    --tasks  scienceqa\
    --output_path "/hdd0/bmy/lmms-eval/evaluation_results"\
    --batch_size 1
```

Hello, I encountered the same issue when testing the ScienceQA dataset. I found that not all data points cause this problem, but rather a specific data entry has an abnormal ‘seq_length’ value during processing, which is even smaller than the sum of ‘system_token_nums’ and ‘image_token_nums’. I was wondering if you have managed to resolve this issue?

Thanks for your reply, i am sorry that i was busy with other projects and have no idea about that. I see that a contributor has taken this on, so hopefully it'll be fixed soon.

Hi @Luodian and sorry to interrupt,
Just a gentle follow-up on my ScienceQA evaluation issue. Still running into the same problem with LLaVA evaluation.
Any chance someone could take a quick look when possible? Happy to provide more details if needed.
Thanks! 🙏

Hi，I've encountered the same question as you have.
I solved it by forcing it to load the 'simple' model instead of 'chat' model

@ lmms-eval-main/lmms_eval/models/__init__.py:109, get_model():
```
    if model_name in AVAILABLE_CHAT_TEMPLATE_MODELS:
        model_type = "chat"
        AVAILABLE_MODELS = AVAILABLE_CHAT_TEMPLATE_MODELS
    else:
        model_type = "simple"
        AVAILABLE_MODELS = AVAILABLE_SIMPLE_MODELS

    #### APPENDED CODE ########
    import os
    if model_name == 'llava_hf' and os.environ['FORCE_SIMPLE_MODEL'] == '1':
        import colorama
        force_simple = True
        print(colorama.Fore.RED + f'@ lmms-eval-main/lmms_eval/models/__init__.py:112 强制加载simple model， force_simple = True, originally False' + colorama.Style.RESET_ALL)
    #### APPENDED CODE END ########
    
    # Override with force_simple if needed, but only if the model exists in AVAILABLE_SIMPLE_MODELS
    if force_simple and model_name in AVAILABLE_SIMPLE_MODELS:
        model_type = "simple"
        AVAILABLE_MODELS = AVAILABLE_SIMPLE_MODELS

```


and bash script as follows:
```
export DATASET_NAME=scienceqa
export FORCE_SIMPLE_MODEL=1

accelerate launch --num_processes=1 -m \
    lmms_eval \
    --model llava_hf   \
    --model_args pretrained=$MODEL_PATH,device_map=""   \
    --tasks $DATASET_NAME  \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix reproduce \
    --output_path ./logs/ 

export FORCE_SIMPLE_MODEL=

```


Noted that it is likely to be VERY SLOW at the progress of 2560+/4241 at a speed of 415s/it. but its not stuck
maybe you can set --num_processes=8 and it will greatly speed up the process