Repository: EvolvingLMMs-Lab/lmms-eval
Issue: MMMU-test doesn't generate submission files
Root cause: Validation Gap

```
python -m lmms_eval \
    --model=internvl2 \
    --model_args=pretrained=my_model_path,device_map=auto \
    --tasks=mmmu_test \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=internvl2 \
    --output_path="./logs/" \
    --limit 100
```
I tried to run MMMU-test on internvl2 model, and i found that the metric in mmmu_test yaml config is 'submission'.
But there is no submission file generated.

And another question is that there is no lmms_eval_specific_kwargs in mmmu_test.yaml, which leads to the following error:

```

> lmms_eval/tasks/mmmu/utils.py", line 145, in mmmu_doc_to_text
>     question = construct_prompt(doc, lmms_eval_specific_kwargs["multiple_choice_prompt"], lmms_eval_specific_kwargs["open_ended_prompt"])
> TypeError: 'NoneType' object is not subscriptable

```

So, should i add my own lmms_eval_specific_kwargs in the yaml config?

i met the same issue too. how do you solve it?

Hi, I replaced 
`question = construct_prompt(doc, lmms_eval_specific_kwargs["multiple_choice_prompt"], lmms_eval_specific_kwargs["open_ended_prompt"])`

with 
`question = construct_prompt(doc, MC_PROMPT, OPEN_ENDED_PROMPT)`

> Hi, I replaced `question = construct_prompt(doc, lmms_eval_specific_kwargs["multiple_choice_prompt"], lmms_eval_specific_kwargs["open_ended_prompt"])`
> 
> with `question = construct_prompt(doc, MC_PROMPT, OPEN_ENDED_PROMPT)`

I'm encountering the same issue now. I added the following code to the related function:
```python
def mmmu_test_aggregate_results_for_submission(results, args):
    eval_logger.info("DEBUG")
    print("DEBUG")
```
But this isn't even being printed to the logs (I used `pip install -e .` and other flags work), so it seems that the function isn't being called:
```
Postprocessing:  81%|████████  | 1057/1313 [00:00<00:00, 2629.42it/s]
Postprocessing: 100%|██████████| 1313/1313 [00:00<00:00, 2626.16it/s]
fatal: 不是 git 仓库（或者任何父目录）：.git
[32m2025-06-13 15:25:25.635[0m | [1mINFO    [0m | [36mlmms_eval.loggers.evaluation_tracker[0m:[36msave_results_aggregated[0m:[36m188[0m - [1mSaving results aggregated[0m
[32m2025-06-13 15:25:25.647[0m | [1mINFO    [0m | [36mlmms_eval.loggers.evaluation_tracker[0m:[36msave_results_samples[0m:[36m255[0m - [1mSaving per-sample results for: mmmu_test[0m
qwen2_5_vl (pretrained=/security/20250612_154355_Qwen2.5-VL-7B-Instruct/v0-20250612-154440/checkpoint-5003,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|-----|-------|------|------|------|---|-----|---|------|
[rank0]:[W613 15:25:31.363949964 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://torch.org/docs/stable/distributed.html#shutdown (function operator())
```
My `mmmu_test.yaml` looks like this:
```
dataset_path: lmms-lab/MMMU
task: "mmmu_test"
test_split: test
output_type: generate_until
doc_to_visual: !function utils.mmmu_doc_to_visual
doc_to_text: !function utils.mmmu_doc_to_text
doc_to_target: "answer"
# The return value of process_results will be used by metrics
process_results: !function utils.mmmu_process_results
metric_list:
  - metric: submission
    aggregation: !function utils.mmmu_test_aggregate_results_for_submission
    higher_is_better: true
include: _default_template_yaml
```
What should I do about this?

> > Hi, I replaced `question = construct_prompt(doc, lmms_eval_specific_kwargs["multiple_choice_prompt"], lmms_eval_specific_kwargs["open_ended_prompt"])`
> > with `question = construct_prompt(doc, MC_PROMPT, OPEN_ENDED_PROMPT)`
> 
> I'm encountering the same issue now. I added the following code to the related function:
> 
> def mmmu_test_aggregate_results_for_submission(results, args):
>     eval_logger.info("DEBUG")
>     print("DEBUG")
> But this isn't even being printed to the logs (I used `pip install -e .` and other flags work), so it seems that the function isn't being called:
> 
> ```
> Postprocessing:  81%|████████  | 1057/1313 [00:00<00:00, 2629.42it/s]
> Postprocessing: 100%|██████████| 1313/1313 [00:00<00:00, 2626.16it/s]
> fatal: 不是 git 仓库（或者任何父目录）：.git
> [32m2025-06-13 15:25:25.635[0m | [1mINFO    [0m | [36mlmms_eval.loggers.evaluation_tracker[0m:[36msave_results_aggregated[0m:[36m188[0m - [1mSaving results aggregated[0m
> [32m2025-06-13 15:25:25.647[0m | [1mINFO    [0m | [36mlmms_eval.loggers.evaluation_tracker[0m:[36msave_results_samples[0m:[36m255[0m - [1mSaving per-sample results for: mmmu_test[0m
> qwen2_5_vl (pretrained=/security/20250612_154355_Qwen2.5-VL-7B-Instruct/v0-20250612-154440/checkpoint-5003,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
> |Tasks|Version|Filter|n-shot|Metric|   |Value|   |Stderr|
> |-----|-------|------|------|------|---|-----|---|------|
> [rank0]:[W613 15:25:31.363949964 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://torch.org/docs/stable/distributed.html#shutdown (function operator())
> ```
> 
> My `mmmu_test.yaml` looks like this:
> 
> ```
> dataset_path: lmms-lab/MMMU
> task: "mmmu_test"
> test_split: test
> output_type: generate_until
> doc_to_visual: !function utils.mmmu_doc_to_visual
> doc_to_text: !function utils.mmmu_doc_to_text
> doc_to_target: "answer"
> # The return value of process_results will be used by metrics
> process_results: !function utils.mmmu_process_results
> metric_list:
>   - metric: submission
>     aggregation: !function utils.mmmu_test_aggregate_results_for_submission
>     higher_is_better: true
> include: _default_template_yaml
> ```
> 
> What should I do about this?

I've submitted a PR to fix this issue https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/724