Repository: EvolvingLMMs-Lab/lmms-eval
Issue: MMMU引发datasets.exceptions.NonMatchingSplitsSizesError
Root cause: External Dependency Breakage

正常配置环境，测试MMMU-val时候引发下面的错误：
Traceback (most recent call last):
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/api/task.py", line 1052, in download
    self.dataset = datasets.load_dataset(
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/datasets/load.py", line 2084, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/datasets/builder.py", line 925, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/datasets/builder.py", line 1019, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/datasets/utils/info_utils.py", line 77, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.exceptions.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='dev', num_bytes=57719107.0, num_examples=150, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='dev', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='mmmu')}, {'expected': SplitInfo(name='test', num_bytes=3271046267.0, num_examples=10500, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='test', num_bytes=2031608167, num_examples=3000, shard_lengths=[600, 900, 500, 1000], dataset_name='mmmu')}]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/__main__.py", line 349, in cli_evaluate
    results, samples = cli_evaluate_single(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/__main__.py", line 484, in cli_evaluate_single
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/utils.py", line 536, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/evaluator.py", line 213, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager, task_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/tasks/__init__.py", line 565, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/tasks/__init__.py", line 378, in load_task_or_group
    all_loaded_tasks = dict(collections.ChainMap(*map(load_fn, task_list)))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/tasks/__init__.py", line 297, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/tasks/__init__.py", line 267, in _load_task
    task_object = TaskObj(config=config, model_name=self.model_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/api/task.py", line 1646, in __init__
    super().__init__(data_dir, cache_dir, download_mode, config, model_name)
  File "/gemini/user/private/MLLM-QAT/3rdparty/lmms-eval/lmms_eval/api/task.py", line 722, in __init__
    self.download(self.config.dataset_kwargs)
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/tenacity/__init__.py", line 330, in wrapped_f
    return self(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/tenacity/__init__.py", line 467, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/qwen/lib/python3.12/site-packages/tenacity/__init__.py", line 411, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7f2b1077b530 state=finished raised NonMatchingSplitsSizesError>]
2025-11-12 08:02:33 | ERROR    | __main__:cli_evaluate:371 - Error during evaluation: RetryError[<Future at 0x7f2b1077b530 state=finished raised NonMatchingSplitsSizesError>]. Please set `--verbosity=DEBUG` to get more information.
/opt/conda/envs/qwen/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/opt/conda/envs/qwen/lib/python3.12/multiprocessing/resource_tracker.py:292: UserWarning: resource_tracker: '/mp-ao5f6ho7': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))



运行命令：
tasks=mmmu_val
model="my/path/to/Qwen3-VL-4B-Instruct"
accelerate launch --num_processes=${ngpu} --main_process_port=12346 -m lmms_eval \
    --model ${model} \
    --model_args=${model_args} \
    --tasks ${tasks} \
    --batch_size 1

请问如何解决这个问题？

你好，我也遇到了相同的问题，请问你怎么解决的呀

I encountered the same issue. How did you resolve it?