Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] MLVU evaluation bug, KeyError: 'needle'
Root cause: Validation Gap

This used to work, but recently I can not do MLVU evaluation anymore.

^MPostprocessing:   0%|          | 0/544 [00:00<?, ?it/s]^MPostprocessing:  69%|██████▉   | 374/544 [00:00<00:00, 3731.27it/s]^MPostprocessing: 100%|██████████| 544/544 [00:00<00:00, 3723.99it/s]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/atuin/b232dd/b232dd16/software/private/conda/envs/llava1/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/home/atuin/b232dd/b232dd16/software/private/conda/envs/llava1/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/__main__.py", line 533, in <module>
[rank0]:     cli_evaluate()
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/__main__.py", line 346, in cli_evaluate
[rank0]:     raise e
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/__main__.py", line 330, in cli_evaluate
[rank0]:     results, samples = cli_evaluate_single(args)
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/__main__.py", line 471, in cli_evaluate_single
[rank0]:     results = evaluator.simple_evaluate(
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator.py", line 243, in simple_evaluate
[rank0]:     results = evaluate(
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator.py", line 579, in evaluate
[rank0]:     task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator_utils.py", line 114, in calculate_aggregate_metric
[rank0]:     self.agg_metrics[metric_key] = agg_fn(items)
[rank0]:   File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/tasks/mlvu/utils.py", line 105, in mlvu_aggregate_results
[rank0]:     category2score[task_type]["answered"] += 1
[rank0]: KeyError: 'needle'


I also get the same error. 

So any ideas about it ?

> I also get the same error.



> This used to work, but recently I can not do MLVU evaluation anymore.
> 
> ^MPostprocessing: 0%| | 0/544 [00:00<?, ?it/s]^MPostprocessing: 69%|██████▉ | 374/544 [00:00<00:00, 3731.27it/s]^MPostprocessing: 100%|██████████| 544/544 [00:00<00:00, 3723.99it/s] [rank0]: Traceback (most recent call last): [rank0]: File "/home/atuin/b232dd/b232dd16/software/private/conda/envs/llava1/lib/python3.10/runpy.py", line 196, in _run_module_as_main [rank0]: return _run_code(code, main_globals, None, [rank0]: File "/home/atuin/b232dd/b232dd16/software/private/conda/envs/llava1/lib/python3.10/runpy.py", line 86, in _run_code [rank0]: exec(code, run_globals) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 533, in [rank0]: cli_evaluate() [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 346, in cli_evaluate [rank0]: raise e [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 330, in cli_evaluate [rank0]: results, samples = cli_evaluate_single(args) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 471, in cli_evaluate_single [rank0]: results = evaluator.simple_evaluate( [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper [rank0]: return fn(*args, **kwargs) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator.py", line 243, in simple_evaluate [rank0]: results = evaluate( [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper [rank0]: return fn(*args, **kwargs) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator.py", line 579, in evaluate [rank0]: task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator_utils.py", line 114, in calculate_aggregate_metric [rank0]: self.agg_metrics[metric_key] = agg_fn(items) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/tasks/mlvu/utils.py", line 105, in mlvu_aggregate_results [rank0]: category2score[task_type]["answered"] += 1 [rank0]: KeyError: 'needle'

I also get the same error. Do you know how to solve it?

> > I also get the same error.
> 
> > This used to work, but recently I can not do MLVU evaluation anymore.
> > ^MPostprocessing: 0%| | 0/544 [00:00<?, ?it/s]^MPostprocessing: 69%|██████▉ | 374/544 [00:00<00:00, 3731.27it/s]^MPostprocessing: 100%|██████████| 544/544 [00:00<00:00, 3723.99it/s] [rank0]: Traceback (most recent call last): [rank0]: File "/home/atuin/b232dd/b232dd16/software/private/conda/envs/llava1/lib/python3.10/runpy.py", line 196, in _run_module_as_main [rank0]: return _run_code(code, main_globals, None, [rank0]: File "/home/atuin/b232dd/b232dd16/software/private/conda/envs/llava1/lib/python3.10/runpy.py", line 86, in _run_code [rank0]: exec(code, run_globals) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 533, in [rank0]: cli_evaluate() [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 346, in cli_evaluate [rank0]: raise e [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 330, in cli_evaluate [rank0]: results, samples = cli_evaluate_single(args) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/**main**.py", line 471, in cli_evaluate_single [rank0]: results = evaluator.simple_evaluate( [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper [rank0]: return fn(*args, **kwargs) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator.py", line 243, in simple_evaluate [rank0]: results = evaluate( [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper [rank0]: return fn(*args, **kwargs) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator.py", line 579, in evaluate [rank0]: task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/evaluator_utils.py", line 114, in calculate_aggregate_metric [rank0]: self.agg_metrics[metric_key] = agg_fn(items) [rank0]: File "/home/hpc/b232dd/b232dd16/lmms-eval/lmms_eval/tasks/mlvu/utils.py", line 105, in mlvu_aggregate_results [rank0]: category2score[task_type]["answered"] += 1 [rank0]: KeyError: 'needle'
> 
> I also get the same error. Do you know how to solve it?

No, it seems that this problem still has not been addressed. I turned to use "mlvu_dev" and "mlvu_test" in the end, this works by lmms-eval. But notice that the score may not be the same as "mlvu".