Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Unable to Conduct Video Evaluation on InternVL2
Root cause: Validation Gap

Hello, I am encountering an issue when attempting to conduct video evaluation on InternVL2. Below is my script and the error details. Notably, image evaluation works as expected, but video evaluation triggers the error shown below. Any guidance or suggestions on resolving this would be greatly appreciated.
Script:：
cd /share/project/lmms-eval
TASK='videomme'
CKPT_PATH='/share/project/OpenGVLab/InternVL2-8B'
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX
accelerate launch --num_processes 8 -m lmms_eval \
    --model internvl2 \
    --model_args pretrained=$CKPT_PATH \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path /share/project/lmms-eval/logs/
    
Error:
    Traceback (most recent call last):
  File "/share/project/lmms-eval/lmms_eval/__main__.py", line 329, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/share/project/lmms-eval/lmms_eval/__main__.py", line 470, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/share/project/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "/share/project/lmms-eval/lmms_eval/evaluator.py", line 243, in simple_evaluate
    results = evaluate(
  File "/share/project/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "/share/project/lmms-eval/lmms_eval/evaluator.py", line 457, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/share/project/lmms-eval/lmms_eval/models/internvl2.py", line 259, in generate_until
    visuals = [load_image(visual).to(torch.bfloat16).cuda() for visual in visuals]
  File "/share/project/lmms-eval/lmms_eval/models/internvl2.py", line 259, in <listcomp>
    visuals = [load_image(visual).to(torch.bfloat16).cuda() for visual in visuals]
Model Responding:   0%|                                                                                                                                                              | 0/338 [00:00<?, ?it/s]
  File "/share/projec/lmms-eval/lmms_eval/models/internvl2.py", line 85, in load_image
    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
  File "/share/project/lmms-eval/lmms_eval/models/internvl2.py", line 53, in dynamic_preprocess
    orig_width, orig_height = image.size
AttributeError: 'str' object has no attribute 'size'
2024-10-29 14:53:30.041 | ERROR    | __main__:cli_evaluate:348 - Error during evaluation: 'str' object has no attribute 'size'. Please set `--verbosity=DEBUG` to get more information.


You also need to pass in `modality=video` in the model_args

> You also need to pass in `modality=video` in the model_args

Thank you for your response; the issue has been resolved!

Additionally, I would like to ask about the evaluation scripts for qwen2_vl and minicpm_v when testing on videos. I tried changing the model parameter to minicpm_v and modified --model_args to pretrained=/share/OpenBMB/MiniCPM-V-2_6, max_frames_num=$MAX_FRAMES, modality=video \, keeping other parameters unchanged, but this resulted in an error. Could you advise on the correct parameter settings or let me know if I missed any necessary configurations?

Thank you very much for your assistance!


Not all model support video evaluation and if you need to evaluate video on minicpm_v you will have to handle the processing logic by yourself. You will need to check the model's init parameters to decide what args you can pass using the `model_args`. Otherwise it won't make any effect.