Repository: EvolvingLMMs-Lab/lmms-eval
Issue: TypeError: 'JpegImageFile' object is not iterable
Root cause: Validation Gap

2025-01-26 16:24:34.358 | ERROR    | __main__:cli_evaluate:349 - Error during evaluation: 'JpegImageFile' object is not iterable. Please set `--verbosity=DEBUG` to get more information.
Token indices sequence length is longer than the specified maximum sequence length for this model (2699 > 2048). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
  File "lmms-eval/lmms_eval/__main__.py", line 330, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/lmms-eval/lmms_eval/__main__.py", line 471, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "lmms-eval/lmms_eval/evaluator.py", line 243, in simple_evaluate
    results = evaluate(
  File "/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "/lmms-eval/lmms_eval/evaluator.py", line 453, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/lmms-eval/lmms_eval/models/llava.py", line 314, in generate_until
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "lmms-eval/lmms_eval/models/llava.py", line 314, in <listcomp>
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "/lmms-eval/lmms_eval/api/task.py", line 1347, in doc_to_visual
    else self.config.doc_to_visual(
  File "lmms-eval/lmms_eval/tasks/mmt/utils.py", line 53, in mmt_doc_to_visual
    return [image.convert("RGB") for image in doc["image"]]
TypeError: 'JpegImageFile' object is not iterable
2025-01-26 16:28:41.128 | ERROR    | __main__:cli_evaluate:349 - Error during evaluation: 'JpegImageFile' object is not iterable. Please set `--verbosity=DEBUG` to get more information.
Token indices sequence length is longer than the specified maximum sequence length for this model (2681 > 2048). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
  File "lmms-eval/lmms_eval/__main__.py", line 330, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "lmms-eval/lmms_eval/__main__.py", line 471, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "lmms-eval/lmms_eval/evaluator.py", line 243, in simple_evaluate
    results = evaluate(
  File "lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "lmms-eval/lmms_eval/evaluator.py", line 453, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/lmms-eval/lmms_eval/models/llava.py", line 314, in generate_until
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "/lmms-eval/lmms_eval/models/llava.py", line 314, in <listcomp>
    batched_visuals = [doc_to_visual[0](self.task_dict[task][split][ids]) for ids in doc_id]  # [B, N]
  File "lmms-eval/lmms_eval/api/task.py", line 1347, in doc_to_visual
    else self.config.doc_to_visual(
  File "lmms-eval/lmms_eval/tasks/mmt/utils.py", line 53, in mmt_doc_to_visual
    return [image.convert("RGB") for image in doc["image"]]
TypeError: 'JpegImageFile' object is not iterable
2025-01-26 16:33:44.349 | ERROR    | __main__:cli_evaluate:349 - Error during evaluation: 'JpegImageFile' object is not iterable. Please set `--verbosity=DEBUG` to get more information.

I meet the same problem.

I meet the same problem when evaluating on the MMT-Bench

I encountered the same problem when evaluating on MMT-Bench. The bug was solved by adjusting the `mmt_doc_to_visual` function:
```
def mmt_doc_to_visual(doc):
    if isinstance(doc["image"], list):
        return [image.convert("RGB") for image in doc["image"]]
    else:
        return [doc["image"].convert("RGB")]
```
I hope this helps someone who is also experiencing this problem.