zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Type error] cache_requests parameter is mismatch.
Root cause: Interface Contract Mismatch


The cache_requests Type in __main__ and the cache_requests Type in simple_evaluation are different.

In main, the type of cache_requests is string type, and in simple_evaluation, the type is boolean. Please check which one is correct.

If the string type used in main is correct, please also explain the meaning of each value.

__main__.py
`    parser.add_argument(
        "--cache_requests",
        type=str,
        default=None,
        choices=["true", "refresh", "delete"],
        help="Speed up evaluation by caching the building of dataset requests. `None` if not caching.",
    )`

evaluator.py

`def simple_evaluate(
    model,
    model_args: Optional[Union[str, dict]] = None,
    tasks: Optional[List[Union[str, dict, object]]] = None,
    num_fewshot: Optional[int] = None,
    batch_size: Optional[Union[int, str]] = None,
    max_batch_size: Optional[int] = None,
    device: Optional[str] = None,
    use_cache: Optional[str] = None,
    cache_requests: bool = False,
    rewrite_requests_cache: bool = False,
    delete_requests_cache: bool = False,
    limit: Optional[Union[int, float]] = None,
    bootstrap_iters: int = 100000,
    check_integrity: bool = False,
    write_out: bool = False,
    log_samples: bool = True,
    evaluation_tracker: Optional[EvaluationTracker] = None,
    system_instruction: Optional[str] = None,
    apply_chat_template: bool = False,
    fewshot_as_multiturn: bool = False,
    gen_kwargs: Optional[str] = None,
    task_manager: Optional[TaskManager] = None,
    verbosity: str = "INFO",
    predict_only: bool = False,
    random_seed: int = 0,
    numpy_random_seed: int = 1234,
    torch_random_seed: int = 1234,
    fewshot_random_seed: int = 1234,
    datetime_str: str = get_datetime_str(),
    cli_args=None,
):`




Hi, I think cache requests has not been tested in this framework and is simply migrated from the original lm-eval.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: load_video gets 0 frame for mp4 file
Root cause: Validation Gap

Hi, when I evaluated on LongVideoBench with the provided command:
```bash
python3 -m accelerate.commands.launch \
    --main_process_port 11223 \
    --num_processes=1 -m lmms_eval \
    --model video_llava \
    --tasks longvideobench_val_v \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix video_llava_lvb_v \
    --output_path ./logs/ \
    --verbosity DEBUG
```
 I got `need at least one array to stack` error when running `lmms_eval/models/model_utils/load_video.py`. I think [this line](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/ce440f81a9e6ff2aa2e35f3dbc7845d4b3cff5ab/lmms_eval/models/model_utils/load_video.py#L51) may cause this error. For some mp4 files, the metadata doesn't contain the frame information. Is it a bug or is there something wrong with my command?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: ValueError: BuilderConfig 'testdev_balanced_instructions' not found. Available: ['default']
Root cause: Configuration Error

When I used the downloaded dataset to evaluate GQA, I got the following errors：


Traceback (most recent call last):
  File "/opt/miniconda/lib/python3.10/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
  File "/apdcephfs_cq11/share_1567347/share_info/mingxiaoli/git/lmms-eval/lmms_eval/api/task.py", line 1047, in download
    self.dataset = datasets.load_dataset(
  File "/opt/miniconda/lib/python3.10/site-packages/datasets/load.py", line 2523, in load_dataset
    builder_instance = load_dataset_builder(
  File "/opt/miniconda/lib/python3.10/site-packages/datasets/load.py", line 2232, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/opt/miniconda/lib/python3.10/site-packages/datasets/builder.py", line 371, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/opt/miniconda/lib/python3.10/site-packages/datasets/builder.py", line 592, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'testdev_balanced_instructions' not found. Available: ['default']

@lmx760581375 Hi!!! Can I take a look at the code you used to run `lmms-eval`?

Hi, have you solved the error? I got the same ValueError. The only modification I made is `dataset_path` in `gqa.yaml`
and here's the script:
```
python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="/home/xxx/LLaVA/playground/model/llava-v1.5-7b" \
    --tasks gqa \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix test \
    --output_path ./logs/
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [OOM] CUDA out of memory error during evaluating llava-hf and llama-vision models
Root cause: Configuration Error

I attempted evaluations using the llava-hf and llama-vision models but encountered out-of-memory (OOM) errors.

For the llava-hf model, the OOM error occurred when using the VQAv2 dataset, while it did not occur when using the MMMU dataset.

For the llama-vision model, OOM errors occurred with all datasets, even when utilizing nine 40GB GPUs. Applying device_map=auto did not resolve the issue.

Is there a way to perform evaluations without encountering OOM errors?

Below is the shell script I used for reference.

`python3 -m accelerate.commands.launch \
    --num_processes=9 \
    -m lmms_eval \
    --model llava_hf \
    --model_args pretrained="llava-hf/llava-v1.6-vicuna-7b-hf" \
    --tasks vqav2 \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_vqav2 \
    --output_path ./logs/
`

`python3 -m accelerate.commands.launch \
    --num_processes=9 \
    -m lmms_eval \
    --model llama_vision \
    --model_args pretrained="meta-llama/Llama-3.2-11B-Vision-Instruct",device_map=auto \
    --tasks mmmu \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llama_11b_mmmu \
    --output_path ./logs/ \
`


Hiii! You may try to start the task directly using `lmms-eval`, because if you haven't run `accelerate config`, `accelerate` will, by default, load one model on each GPU:

```sh
lmms-eval \
    --model llama_vision \
    --model_args pretrained="meta-llama/Llama-3.2-11B-Vision-Instruct",device_map=auto \
    --tasks mmmu \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llama_11b_mmmu \
     --output_path ./logs/ 
```

I have the same problem, which occured at Qwen2VL
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: reproduce the LLaVA-NeXT-Video 7B scores
Root cause: Documentation Deficiency

reference: https://lmms-lab.github.io/posts/lmms-eval-0.2/,

In the section of "Alignment Check for Video Datasets", there are LLaVA-Next-Video-7B scores in the benchmark table.
Could you share the corresponding model checkpoint link? I'm very confused  with the naming of version..
1. https://huggingface.co/lmms-lab/LLaVA-NeXT-Video-7B
2. https://huggingface.co/lmms-lab/LLaVA-Video-7B-Qwen2

And also the scripts for evaluation?

i have the same issue. Difficult to understand exactly which checkpoint and model_name to use to reproduce those results
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: long time waiting after postprocessing
Root cause: Algorithmic Error

![Image](https://github.com/user-attachments/assets/c7104c25-7e2d-41f7-8a24-da1507ab34d6)

it's long time for waiting after postprocessing.

I'm evaluating seedbench_lite.

It's because saving the samples take long time, maybe seedbench's `doc` contains too much information. 

You could try to remove `--log_samples` to see if that accelerate the log.

But for long run, you may need to hack to the lines to make sure the results are not too big.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/16f2c6bd1d76cbe720b4bab3258476298a156c89/lmms_eval/loggers/evaluation_tracker.py#L198-L205
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2-VL,IndexError: index 0 is out of bounds for dimension 0 with size 0
Root cause: Configuration Error

I want to test Qwen2-VL model on OCRbench, but I got this error . How can i fix it.
my commands are as follows:

```
python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model qwen2_vl \
    --model_args pretrained="Qwen/Qwen2-VL-7B"  \
    --tasks ocrbench \
    --batch_size 1 \
    --output_path ./logs/ \
    --verbosity=DEBUG

```

```
File"/envs/lmms-eval/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1770
in prepare_inputs_for_generation
    if cache_position[0] != 0:
IndexError: index 0 is out of bounds for dimension 0 with size 0
```

same question, how do you fix it

same issue,have  you solved thie question

@iAsakiT3T @ZizhenWang So sorry，I forgot whether I solved this，I'm currently using VLMEvalKit for evaluation.

I encountered the same problem, I found the solution from another [thread](https://github.com/QwenLM/Qwen3-VL/issues/596#issuecomment-2576894270):
> I encountered the same problem. The root cause is that the chat_template of Qwen2-VL and Qwen2-VL-Instruct is different, which causes the input_ids to be empty before inference. So the solution is to replace the chat_template.json of Qwen2-VL with the chat_template.json of Instruct. It solved my problem.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How do args flow into the LLava model after evaluate() in evaluator.py?

Thank you for this fantastic project! 

I’m trying to understand how command-line arguments (args) are passed into the LLava model. I’ve traced the parameter flow and found that the arguments appear as cli_args in the evaluate() function in evaluator.py. However, after this point, I’m unsure where the arguments go and how they are eventually passed into the LLava model.

Could you kindly provide some guidance or point me to the relevant part of the code? Thanks a lot for your help!

I think it is being passed in this line through model_args

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/f6fe367ceb579f84eeca0e251689f8c1024444bb/lmms_eval/evaluator.py#L176-L183

> I think it is being passed in this line through model_args
> 
> [lmms-eval/lmms_eval/evaluator.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/f6fe367ceb579f84eeca0e251689f8c1024444bb/lmms_eval/evaluator.py#L176-L183)
> 
> Lines 176 to 183 in [f6fe367](/EvolvingLMMs-Lab/lmms-eval/commit/f6fe367ceb579f84eeca0e251689f8c1024444bb)
> 
>  lm = lmms_eval.models.get_model(model).create_from_arg_string( 
>      model_args, 
>      { 
>          "batch_size": batch_size, 
>          "max_batch_size": max_batch_size, 
>          "device": device, 
>      }, 
>  )

Thank you, this is very helpful!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue:  different result between lmms-eval and llava-eval on text-vqa
Root cause: Configuration Error

I used the lmms-eval repo to evaluate my model and got the following result:
<img width="434" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/35927125/924d5860-e977-49a6-b8a0-3b194ccf2d14">

So I used the evaluation script in llava and got the following result:
<img width="1263" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/35927125/e0065657-fe1a-426f-a489-45e988cb7c99">

Why are they so different?


Hi, you can refer to #6

> Hi, you can refer to #6

Thanks for replying!

However, you can see the number of samples; it is the same as the number of samples in the validation split. And I am sure I used the validation split to evaluate the model.

By the way, the dataset of science qa also have same issue.

Hi @yiyexy , there is one more comment below in #6 addressing the real problem of causing the inconsistency. It is because whether you use ocr or not.

For scienceqa, may I ask what is your score? I think our results are quite similar to the reported results. You can check the full list of our results [here](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit#gid=0)

@kcz358 Ok, I will try it later.
I found that there is more than one point difference in the scienceqa scores. Why?
I got 72.83 by using lmms-eval, but only got 70.90 with the llava script.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Running LLaVA-NeXT - Vicuna 7B
Root cause: Documentation Deficiency

Hello, 

I am trying to run  LLaVA-NeXT - Vicuna 7B on the textcaps task. I use the command below after running the installation steps for LLaVA-NeXT as outlined in the readme of this repo:

```
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="llava-hf/llava-v1.6-vicuna-7b-hf" \
    --tasks textcaps \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_next_textcaps \
    --output_path ./logs/
```

I get the following error:
```
You are using a model of type llava_next to instantiate a model of type llava_llama. This is not supported for all configurations of models and can yield errors.
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.                                                                                                          
Traceback (most recent call last):                                                                          
  File "/home/user/lmms-eval/lmms_eval/models/llava.py", line 99, in __init__                       
    self._tokenizer, self._model, self._image_processor, self._max_length = load_pretrained_model(pretrained
, None, model_name, device_map=self.device_map, **llava_model_args)                                         
  File "/home/user/lmms-eval/LLaVA/llava/model/builder.py", line 117, in load_pretrained_model      
    model = LlavaLlamaForCausalLM.from_pretrained(                                                          
  File "/home/user/.conda/envs/lmms-eval/lib/python3.10/site-packages/transformers/modeling_utils.py
", line 3594, in from_pretrained                                                                            
    model = cls(config, *model_args, **model_kwargs)                                                        
TypeError: LlavaLlamaForCausalLM.__init__() got an unexpected keyword argument 'multimodal'
```
I am not sure what to do to run this baseline.

Please use `llava-hf` instead of `llava`. Thanks!

Hello, thanks a lot. It works now after I fixed one line of code in the LlavaHf class (lmms-eval/lmms_eval/models/llava_hf.py) for single GPU evaluation (line 89):
`self.device_map = 'auto'` 
instead of
`self.device_map = device_map`

It is a hacky fix, but it works!

The command I am using now is:
```
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava_hf \
    --model_args pretrained="llava-hf/llava-v1.6-vicuna-7b-hf" \
    --tasks textcaps \
    --batch_size 1 \
    --log_samples \
    --device cuda:0 \
    --log_samples_suffix llava_next_textcaps-prompt-0 \
    --output_path ./logs/ 
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Performance] Evaluation not terminating
Root cause: Resource Mishandling

Hi,
When evaluating Qwen2VL model on MLVU on 4 GPUs, even after all samples processed, final evaluation is not printed. 2 GPUs processes are not terminated.

![Image](https://github.com/user-attachments/assets/0199f7ee-f7c6-47f5-983e-237857be3384)

![Image](https://github.com/user-attachments/assets/814a4a3b-88b3-4f14-b210-cdefd6957ce5)


```
export DECORD_EOF_RETRY_MAX=20480

accelerate launch --num_processes 4 --main_process_port 12331 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=/scratch/ykulka10/ms-swift/output/v8-20250201-171811/checkpoint-625-merged \
    --tasks mlvu \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix videomme_tpo \
    --output_path /scratch/ykulka10/lmms-eval/logs 
```

Same problem. Did you solve it?
By the way, do you know how to speed the inference process? Qwen2-VL is so slow...
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: MLVU aggregate function is incorrect
Root cause: Algorithmic Error

According to results in their [leaderboard](https://github.com/JUNJIE99/MLVU?tab=readme-ov-file#trophy-mlvu-test-leaderboard), the overall score is calculated by the average of each subtype accuracy, not the sample accuracy.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d294646d80d5df4f56ae1a057146e09692388cb9/lmms_eval/tasks/mlvu/utils.py#L117-L122
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: MLVU aggregate function is incorrect
Root cause: Algorithmic Error

According to results in their [leaderboard](https://github.com/JUNJIE99/MLVU?tab=readme-ov-file#trophy-mlvu-test-leaderboard), the overall score is calculated by the average of each subtype accuracy, not the sample accuracy.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d294646d80d5df4f56ae1a057146e09692388cb9/lmms_eval/tasks/mlvu/utils.py#L117-L122

Hi @shuyansy , do you mind take a look whether this logic is correct? Thanks!

Hi @SCZwangxiao , this issue should have been fixed in #555
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: AttributeError: 'dict' object has no attribute 'to_dict'
Root cause: External Dependency Breakage

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model llava_vid \
    --model_args pretrained=lmms-lab/LLaVA-NeXT-Video-32B-Qwen,conv_template=qwen_1_5,video_decode_backend=decord,max_frames_num=32,mm_spatial_pool_mode=average,mm_newline_position=grid,mm_resampler_location=after \
    --tasks videomme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_vid_32B \
    --output_path ./logs/


Running the newly updated cli code gives the error AttributeError: 'dict' object has no attribute 'to_dict'

Excuse me, have you solved the problem? 

> Excuse me, have you solved the problem?

Oh, never mind. I downgraded the transformers version from 4.49.0 to 4.39.3 to solved it.

but if I downgraded the transformers to 4.39.0, the transformers can not recognize llava_onevision

> The checkpoint you are trying to load has model type `llava_onevision` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

if I use the version >= 4.45.0, the same issue like you again
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Use --use_cache but the responses are not saved
Root cause: Validation Gap

Thank you for your excellent contribution in lmms evaluation, I met a problem when saving the response when evaluate lmms on my curated dataset. The code was executed correctly except for there was no cache files in the folder I initiated in command:
I run:
`python3 -m accelerate.commands.launch     --num_processes=1     -m lmms_eval     --model qwen2_vl     --model_args pretrained="path/to//Qwen2-VL-7B-Instruct"     --tasks dgqa     --batch_size 1     --log_samples     --log_samples_suffix Qwen2-VL-7B-Instruct_dgqa     --output_path ./logs/ --limit 10 --use_cache ./cache/qwen_dgqa_`
dgqa is my new task. While all other procedues(response and evaluation) could be executed correctly, there was no cache file in `./cache/qwen_dgqa_` folder. Please let me know how I can save this problem, thank you very much!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: gen_kwargs['until'] defaults to "\n\n"
Root cause: Configuration Error

In api/task.py, "until" is set to the default value of fewshot_delimiter--"\n\n"
```python
self.generation_kwargs = {
    "until": None if self.fewshot_delimiter is None else [self.fewshot_delimiter],
    "do_sample": False,
}
```
And in models/*.py, the actually default value - eos token of the model, is replaced by "\n\n", which causes issue especially in caption tasks when the output may contain "\n\n"
```python
# qwen2_vl.py for example
# Set default values for until and max_new_tokens
until = [self.tokenizer.decode(self.eot_token_id)]
# Update values from gen_kwargs if present
if "until" in gen_kwargs:
    until = gen_kwargs.pop("until")
    if isinstance(until, str):
        until = [until]
    elif not isinstance(until, list):
        raise ValueError(f"Expected `gen_kwargs['until']` to be of type Union[str,list] but got {type(until)}")
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Request] Support for MileBench
Root cause: Unimplemented Feature Gap

[Song et al., "MileBench: Benchmarking MLLMs in Long Context," COLM-2024.](https://arxiv.org/abs/2404.18532)

MileBench is a benchmark designed for evaluating multimodal large language models (MLLMs) in long sequence inference.
It appears to be frequently used in various papers, but currently, it is not supported.
It would be great to have support for this benchmark.

Looking forward to it too!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to use LLM as judge?
Root cause: Documentation Deficiency

Hi, thank you for your wonderful work! I wonder is there any way that i can use an LLM, for example gpt4o, as the judge for the answer. Thank you very much!


Yes, you can refer to examples such as llava-in-the-wild and llava-wilder for reference. Thank you!

@kcz358  如何全局设置 use llm judge,如果在每个数据集yaml中设置 use llm judge,这样很不便! 如何快速使用llm judge?

大部分需要设置llm as judge的数据集只需要 export OPENAI API 和 OPENAI BASE URL 就可以，如果没有 llm as judge 的逻辑的话需要自行完成 eval 逻辑，由于每个任务大家需要的 eval 逻辑都不一样，很难有一个完全统一的，也可以使用我们提供的统一 llm as judge 接口