Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Coco caption evaluation error
Root cause: Validation Gap

When I use lmms_eval to evaluate coco caption, the following situation occurs
Postprocessing:  99%|█████████▉| 10088/10168 [00:04<00:00, 2586.11it/s]
Postprocessing: 100%|██████████| 10168/10168 [00:04<00:00, 2436.51it/s]
[32m2025-04-25 23:29:19.586[0m | [1mINFO    [0m | [36mutils[0m:[36mcoco_test_aggregation_result[0m:[36m180[0m - [1mStoring prediction that can be submitted to the server ...[0m
[32m2025-04-25 23:29:19.805[0m | [1mINFO    [0m | [36mutils[0m:[36mcoco_test_aggregation_result[0m:[36m184[0m - [1mYour test result has been stored in to /share/user/logs/submissions/coco_captions_test2014_alg_results.json. Make sure you also have the val result stored to submit to the server on https://codalab.lisn.upsaclay.fr/competitions/7404#participate.[0m
[32m2025-04-25 23:29:19.821[0m | [1mINFO    [0m | [36mutils[0m:[36mcoco_test_aggregation_result[0m:[36m180[0m - [1mStoring prediction that can be submitted to the server ...[0m
[32m2025-04-25 23:29:20.039[0m | [1mINFO    [0m | [36mutils[0m:[36mcoco_test_aggregation_result[0m:[36m184[0m - [1mYour test result has been stored in to /share/user/logs/submissions/coco_captions_test2014_alg_results.json. Make sure you also have the val result stored to submit to the server on https://codalab.lisn.upsaclay.fr/competitions/7404#participate.[0m
fatal: Not a git repository (or any parent up to mount point /share)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[32m2025-04-25 23:29:20.063[0m | [1mINFO    [0m | [36mlmms_eval.loggers.evaluation_tracker[0m:[36msave_results_aggregated[0m:[36m188[0m - [1mSaving results aggregated[0m
[32m2025-04-25 23:29:20.110[0m | [1mINFO    [0m | [36mlmms_eval.loggers.evaluation_tracker[0m:[36msave_results_samples[0m:[36m255[0m - [1mSaving per-sample results for: coco2014_cap_test[0m
[32m2025-04-25 23:29:22.100[0m | [1mINFO    [0m | [36mlmms_eval.loggers.evaluation_tracker[0m:[36msave_results_samples[0m:[36m255[0m - [1mSaving per-sample results for: coco2017_cap_test[0m
internvl2 (pretrained=/share/user/output/InternVL2-8B/v1-20250421-175532/checkpoint-8500-merged), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|       Tasks        |Version|Filter|n-shot|     Metric     |   |Value|   |Stderr|
|--------------------|-------|------|-----:|----------------|---|-----|---|------|
|coco_cap            |    N/A|      |      |                |   |     |   |      |
| - coco2014_cap_test|Yaml   |none  |     0|coco_passthrough|↑  |N/A  |±  |   N/A|
| - coco2017_cap_test|Yaml   |none  |     0|coco_passthrough|↑  |N/A  |±  |   N/A|

[rank0]:[W425 23:29:25.206460402 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

Following are the commands to run：
HF_DATASETS_OFFLINE=1
CKPT_PATH=/share/user/modelscope/output/InternVL2-8B/v1-20250421-175532/checkpoint-8500-merged
CUDA_VISIBLE_DEVICES=0,1,2,3 
accelerate launch --num_processes 4 --main_process_port 12380 -m lmms_eval \
    --model internvl2 \
    --model_args pretrained=$CKPT_PATH \
    --tasks coco_cap \
    --batch_size 1 \
    --log_samples \
    --output_path ./logs/

Thanks for your work!

Hi, this is passthrough because we don't have any ground truth for test set I think. You may need to submit to the website for eval

Sorry, I used the wrong dataset. I was using the local dataset coco_karpathy_test.
The minicpmv2.6 
|coco_cap             |    N/A|      |      |            |   |      |   |      |
| - coco_karpathy_test|Yaml   |none  |     0|coco_Bleu_1 |↑  |0.0145|±  |   N/A|
| - coco_karpathy_test|Yaml   |none  |     0|coco_Bleu_2 |↑  |0.0111|±  |   N/A|
| - coco_karpathy_test|Yaml   |none  |     0|coco_Bleu_3 |↑  |0.0083|±  |   N/A|
| - coco_karpathy_test|Yaml   |none  |     0|coco_Bleu_4 |↑  |0.0062|±  |   N/A|
| - coco_karpathy_test|Yaml   |none  |     0|coco_CIDEr  |↑  |0.0000|±  |   N/A|
| - coco_karpathy_test|Yaml   |none  |     0|coco_METEOR |↑  |0.0748|±  |   N/A|
| - coco_karpathy_test|Yaml   |none  |     0|coco_ROUGE_L|↑  |0.2079|±  |   N/A|
The Internvl2-8B
|---------------------|-------|------|-----:|------------|---|-----:|---|------|
|coco_cap | N/A| | | | | | | |
| - coco_karpathy_test|Yaml |none | 0|coco_Bleu_1 |↑ |0.0157|± | N/A|
| - coco_karpathy_test|Yaml |none | 0|coco_Bleu_2 |↑ |0.0122|± | N/A|
| - coco_karpathy_test|Yaml |none | 0|coco_Bleu_3 |↑ |0.0092|± | N/A|
| - coco_karpathy_test|Yaml |none | 0|coco_Bleu_4 |↑ |0.0069|± | N/A|
| - coco_karpathy_test|Yaml |none | 0|coco_CIDEr |↑ |0.0000|± | N/A|
| - coco_karpathy_test|Yaml |none | 0|coco_METEOR |↑ |0.0764|± | N/A|
| - coco_karpathy_test|Yaml |none | 0|coco_ROUGE_L|↑ |0.2167|± | N/A|
The score is too low, and I also checked the generated caption, and I found that the generated description is not very different from the reference caption.

> 抱歉，我使用了错误的数据集。我使用的是本地数据集 coco_karpathy_test。minicpmv2.6 | coco_cap | N/A| | | | | | | | | - coco_karpathy_test|Yaml |none | 0|coco_Bleu_1 |↑ |0.0145|± | N/A| | - coco_karpathy_test|Yaml |none | 0|coco_Bleu_2 |↑ |0.0111|± | N/A| | - coco_karpathy_test|Yaml |none | 0|coco_Bleu_3 |↑ |0.0083|± | N/A| | - coco_karpathy_test|Yaml |none | 0|coco_Bleu_4 |↑ |0.0062|± | N/A| | - coco_karpathy_test|Yaml |none | 0|coco_CIDEr |↑ |0.0000|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_METEOR |↑ |0.0748|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_ROUGE_L|↑ |0.2079|± | N/A| The Internvl2-8B |---------------------|-------|------|-----:|------------|---|-----:|---|---|---| |coco_cap | N/A| | | | | | | | | - coco_karpathy_test|Yaml |无 | 0|coco_Bleu_1 |↑ |0.0157|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_Bleu_2 |↑ |0.0122|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_Bleu_3 |↑ |0.0092|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_Bleu_4 |↑ |0.0069|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_CIDEr |↑ |0.0000|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_METEOR |↑ |0.0764|± | N/A| | - coco_karpathy_test|Yaml |无 | 0|coco_ROUGE_L|↑ |0.2167|± | N/A| 分数太低，并且我还检查了生成的标题，发现生成的描述与参考标题没有太大区别。

Have you solved this problem? I met the same problem now XD.

Hi！
I eventually realized the problem was a format handling error that occurred when I converted the dataset from TSV to JSON and back. I did this to update the links in the file to point to my local dataset.
I suggest you check the contents of your dataset file to see if the issue is caused by several ground truths being merged together (that was the reason for my problem). Alternatively, you can examine the results each time and visually inspect for obvious errors to determine whether it's a dataset issue or a model output issue.
Hope it's useful.