zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Bug?] Doc fields with "image" not saved to jsonl file when using log_samples=True
Root cause: Architectural Constraint

When setting `log_samples=True`, the doc, target, model output, and other info are saved to a jsonl file.

With this line:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d4383329aeaa6ffbcde94a9b31ca0eff7fee557c/lmms_eval/evaluator.py#L525

in the `evaluator.py`, any fields with "image" in the doc key are filtered out. Any reason for that?


Code block:
```python
if log_samples:
      target = task.doc_to_target(doc)
      saved_doc = {}
      for key, value in doc.items():
          # If image is not in key
          if "image" not in key:
              # If audio is also not the value
              if isinstance(value, dict) and "array" in value:
                  continue
              else:
                  saved_doc[key] = value
      filtered_arguments = []
      for req in requests:
          # check if req.args is a list of tuples, and each item in the list is a serializable object
          for value in req.args:
              if isinstance(value, (str, int, float, bool, list, dict, type(None))):
                  filtered_arguments.append(value)
              # else:
              #     filtered_arguments.append(_handle_non_serializable(value))

      example = {
          "doc_id": doc_id,
          "doc": saved_doc,
          "target": target,
          "arguments": filtered_arguments,
          "resps": [req.resps for req in requests],
          "filtered_resps": [req.filtered_resps[filter_key] for req in requests],
          "doc_hash": hash_string(
              json.dumps(
                  requests[0].doc,
                  indent=2,
                  default=handle_non_serializable,
                  ensure_ascii=False,
              )
          ),
          "prompt_hash": hash_string(requests[0].arguments[0]),
          "target_hash": hash_string(str(target)),
      }
      example.update(metrics)
      task_output.logged_samples.append(example)
```


Yes actually in our original design, the image is not supposed to be saved since it can be viewed in the huggingface. Do you think it should be saved locally? Wish to see your thoughts and design on this feature!

I think it is valuable to keep the record that links images to questions. The most useful usecase of the log files is to parse the model responses. In some cases, users may want to check the input image for debugging or visualization. In my case, my HF dataset would download the image files and meta data containing the relative path of images. The dataset also returns meta info such as image size and other labels that may be used in model responses parsing.

The current funtion remove dict from the log files. It would be nice to support dictionary saving in the log files, although flatting the dict is a workaround.

I guess a better solution is to move thoses doc fields filtering and let the users decide whether remove some record in the log files if they want to hide some info.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Is it possible to add support for llava-hf/llava-onevision-qwen2-7b-ov-hf?
Root cause: Documentation Deficiency

Currently, only llava-ov-qwen in lmms-lab format is supported.

I think it's already supported if you use `--model=llava_hf`. Is there any errors? 

Thanks, I used --model=llava_hf, it is worked.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: vLLM Eval Not Working

When running the sample script for vLLM, I get this error:
```
Traceback (most recent call last):
  File "/home/shang/lmms-eval/lmms_eval/__main__.py", line 335, in cli_evaluate
    results, samples = cli_evaluate_single(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shang/lmms-eval/lmms_eval/__main__.py", line 479, in cli_evaluate_single
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shang/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/shang/lmms-eval/lmms_eval/evaluator.py", line 181, in simple_evaluate
    lm = lmms_eval.models.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shang/lmms-eval/lmms_eval/api/model.py", line 110, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/shang/lmms-eval/lmms_eval/models/vllm.py", line 63, in __init__
    self.client = LLM(
                  ^^^
NameError: name 'LLM' is not defined. Did you mean: 'VLLM'?
2025-06-23 12:08:26.497 | ERROR    | __main__:cli_evaluate:357 - Error during evaluation: name 'LLM' is not defined. Please set `--verbosity=DEBUG` to get more information.
```

Is there a fix for this?


Hello! I am confused about it too right now since it seems like it is a bug from EvolvingLLMs team. Have you figured out what's wrong with this and how to fix it (or ignore it)?

Me too, have u fixed it?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Support for MiMo-VL-7B
Root cause: Unimplemented Feature Gap

If I want to evaluate MiMo-VL-7B-SFT, do I just need to use the Qwen2.5-VL model configuration? 
I tried it once and found that the evaluation was very slow and the final result was 0, it must be something wrong with it.
Will it support MiMo？

hi Can you check the repo posted by Mimo's team? 

https://github.com/XiaomiMiMo/lmms-eval

Or you could use `vllm` as inference backend, since it supports most existing models (as long Mimo supports it).

> hi Can you check the repo posted by Mimo's team?
> 
> https://github.com/XiaomiMiMo/lmms-eval
> 
> Or you could use `vllm` as inference backend, since it supports most existing models (as long Mimo supports it).

https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/751  哈喽，我尝试使用vllm 作为后端进行评测，但是发现了这个情况，具体是什么原因呢

> If I want to evaluate MiMo-VL-7B-SFT, do I just need to use the Qwen2.5-VL model configuration? I tried it once and found that the evaluation was very slow and the final result was 0, it must be something wrong with it. Will it support MiMo？

Hi, thanks for your interest. Please adopt our official evaluation framework for reproducible evaluation of MiMo-VL.  https://github.com/XiaomiMiMo/lmms-eval
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: cli_evaluate has option to pass args for use within a module but this does not work
Root cause: Validation Gap

In the main call of the evaluate function (https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/__main__.py#L273), there is the option to directly pass a `argparse.Namespace` to the method which would enable it to be callable from a different python module and not only from the command line.

This does not work however, since the next call directly check the `sys.argv`, so when passing the `args` as a `Namespace` to the function and not via the command line this length check fails. Additionally, there is no handling of the default values if only a subset of the args is passed via a Namespace for the method.

Propossed Solution:
```
def cli_evaluate(args: Union[argparse.Namespace, None] = None) -> None:
    default_args = parse_eval_args()
    
    if args is None and len(sys.argv) == 1:
        print("┌───────────────────────────────────────────────────────────────────────────────┐")
        print("│ Please provide arguments to evaluate the model. e.g.                          │")
        print("│ `python evaluation.py  --model lusxvr/nanoVLM-450M --tasks mmstar`            │")
        print("└───────────────────────────────────────────────────────────────────────────────┘")
        sys.exit(1)

    # If args were provided, override the defaults
    if args:
        for key, value in vars(args).items():
            setattr(default_args, key, value)
    
    args = default_args
```

I've put the fix in this PR, could you help to check? 

#733
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: how to download the dataset correctly?
Root cause: Configuration Error

I use this bash to run an evaluation
```
python -m accelerate.commands.launch\
        --num_processes=1 \
        -m lmms_eval \
        --model llava_onevision \
        --model_args pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwen \
        --tasks mvbench \
        --batch_size 1 \
        --log_samples \
        --log_samples_suffix $RUN_NAME \
        --output_path $LOG_DIR/$RUN_NAME
```
It download dataset into ~/.cache/huggingface/mvbench_video/datasets--OpenGVLab--MVBench/, include files named blobs, refs and snapshots. 

But when evaluating, lmms-eval need video files in '~/.cache/huggingface/mvbench_video/vlnqa/left/*.mp4', thus throwing errors as `Video path: ~/.cache/huggingface/mvbench_video/vlnqa/left/6550_frame53.mp4 does not exist, please check.`

@Sugar929 were you able to solve it? I have the same issue.

@soumyasj I haven't solve this problem, do you have any ideas?

@Sugar929 no, haven't found anything that helps :(

I think the download may not run correctly but you still have the options to download it manually from the repo.

https://huggingface.co/datasets/OpenGVLab/MVBench/tree/main/video

I have the same issues, is there any solution now?

The reason seems to be the `snapshot_download` in `lmms_eval/api/task.py` function wrongly use the arg `cache_dir`, and the old versions do not do like this. The `cache_dir` is designed to save the unzipped videos instead of huggingface dataset repo. The [Commit e24a7d8](https://github.com/EvolvingLMMs-Lab/lmms-eval/commit/e24a7d8ecd2d1c29ded2ac4b85167bc94e6d52ff) seems to fix this already.


I fixed this issue by setting the dataset_kwargs "force_unzip" as 'True' in the yaml file of each dataset, which is defaulted as "False" in task.py.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to download dataset correctly?
Root cause: Configuration Error

The data dir saved is :
~/.cache/huggingface/mvbench_video/datasets--OpenGVLab--MVBench/
include:
blobs  mvbench_video  refs  snapshots
but the framework would read dataset with different path:
like:
~/.cache/huggingface/mvbench_video/tvqa/video_fps3_hq_segment/castle_s05e16_seg02_clip_21_20.26_51.31.mp4
how to solve this issue?

It occurred when downloading mvbench,videomme and egoscheme


anyway to solve this?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Support for llava-hf/llava-v1.6-x
Root cause: Unimplemented Feature Gap

Hi,

I'm trying to run the lmms-eval on llava-hf/llava-v1.6-x and receive an error:
```
  File "/data/long_phan/short-circuiting/lmms-eval/lmms_eval/models/llava_hf.py", line 310, in generate_until
    text_outputs = self.tokenizer.batch_decode(cont, skip_special_tokens=True)[0]
IndexError: list index out of range
```
tracing back I think it is because the ` llava-hf/llava-v1.6` use `LlavaNextForConditionalGeneration` and `LlavaNextProcessor` which does not have `image_sizes`
```
ValueError: The following `model_kwargs` are not used by the model: ['image_sizes'] (note: typos in the generate arguments will also show up in this list)
```

Here is the command that I used:
```
accelerate launch -m lmms_eval --model llava_hf --model_args pretrained=llava-hf/llava-v1.6-mistral-7b-hf  --tasks llava_in_the_wild --batch_size 1 --output_path ./logs/ #
```

Hi @justinphan3110cais , I have fixed the error. You can pull the main again and try the same command.

@kcz358 , Thanks, I saw your fix. However, I think a more robust fix is we can check for the `architectures` in the `AutoConfig`? For example current fix will fail if I have my local version of the llava-hf-1.6 or some model trained from llava-hf-1.6?

I think both are kind of a hard code fix but maybe checking architectures are more flexible. For now I think you just need to make sure your pretrained path contains `1.6` or `1.5` to separate different versions of llava-hf. And I think the way I parse the results also have some problem. Maybe I will try to address these issues later but if you already have your own fix on it you are welcome to raise a PR.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Example Script for llava-hf.
Root cause: Documentation Deficiency

Hi, thanks for your great work! Could you please offer an example script for `llava-hf/llava-1.5-7b-hf`?
I have used the following command, but it throws an error `| ERROR    | lmms_eval.models.llava_hf:generate_until:375 - Error Image features and image tokens do not match: tokens: 0, features 2359296 in generating`.
Here is the script

```
accelerate launch --num_processes=1 -m lmms_eval --model llava_hf \
    --model_args pretrained="llava-hf/llava-1.5-7b-hf",device_map=cuda \
    --tasks "mme" \
    --batch_size 1 --log_samples --output_path "logs" --verbosity=DEBUG
```

Thanks again for your work!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: when to support lmms-lab/MMMU interleaved_format evaluation?
Root cause: Unimplemented Feature Gap


Hi, this is now supported in the new v0.4 release you can check the changes [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/releases/tag/v0.4)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: An issue during evaluating MathVista
Root cause: External Dependency Breakage

I don't know how to solve it, seem to be a problem with huggingface evaluate module.
Traceback (most recent call last):
  File "/home/aka/anaconda3/envs/lmms/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/akaanaconda3/envs/lmms/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/aka/lmms-eval/lmms_eval/__main__.py", line 24, in <module>
    from lmms_eval import evaluator, utils
  File "/home/aka/lmms-eval/lmms_eval/evaluator.py", line 17, in <module>
    import lmms_eval.api.metrics
  File "/home/aka/lmms-eval/lmms_eval/api/metrics.py", line 146, in <module>
    exact_match = evaluate.load("exact_match")
  File "/home/aka/anaconda3/envs/lmms/lib/python3.10/site-packages/evaluate/loading.py", line 751, in load
    evaluation_cls = import_main_class(evaluation_module.module_path)
  File "/home/aka/anaconda3/envs/lmms/lib/python3.10/site-packages/evaluate/loading.py", line 76, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/aka/anaconda3/envs/lmms/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/aka/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/009c8b5313309ea5b135d526433d5ee76508ba1554cbe88310a30f85bb57ec88/exact_match.py", line 16
    }
    ^
SyntaxError: closing parenthesis '}' does not match opening parenthesis '(' on line 14
E0507 14:41:36.009469 139918226703296 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 36925) of binary: /home/aka/anaconda3/envs/lmms/bin/python


You should replace the `exact_match.py` file with this [file](https://github.com/huggingface/evaluate/blob/main/metrics/exact_match/exact_match.py). I encountered the same issue today. It seems to be a problem from hf-mirror.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The results of evaluation on multi-GPUs versus single-GPU differ from each other
Root cause: Resource Mishandling

Thank you for creating this amazing repo!!! It's really helpful!

However, I found an issue when running evaluation on multiple GPUs.
When I evaluate mmbench_en on a single GPU versus multiple GPUs, the results differ from each other.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: 两种评测方式结果差别很大是怎么回事。
Root cause: Configuration Error

如采用intnernvl3-1b-instruct ，在docvqa, mustard, ocrbench, textvqa 上进行测试。
**方式 1.** 
启动服务: CUDA_VISIBLE_DEVICES=6,7 vllm serve /vlm/pretrain_models/OpenGVLab/InternVL3-1B --dtype auto --port 8000   --gpu_memory_utilization 0.4 --tensor-parallel-size 2 --pipeline-parallel-size 1
python3 -m lmms_eval \
    --model openai_compatible \
    --model_args "api_base=http://localhost:8000,api_key=EMPTY,model_version=/vlm/pretrain_models/OpenGVLab/InternVL3-1B" \
    --tasks textvqa,docvqa,ocrbench,mmstar\
    --batch_size 1 \
    --log_samples \
    --output_path ./logs/2025.07.02 \
    --limit 100

**方式 2.**
export CUDA_VISIBLE_DEVICES=0,1,2,3
TASK=$1
CKPT_PATH=$2
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 4 --main_process_port 12380 -m lmms_eval \
    --model internvl2 \
    --model_args pretrained=$CKPT_PATH \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./2025.07.02/ \

<img width="1280" height="266" alt="Image" src="https://github.com/user-attachments/assets/e392dcdd-7376-4ea5-87ad-2977f1a6ce81" />

<img width="1096" height="244" alt="Image" src="https://github.com/user-attachments/assets/1a3d582a-cb62-41a6-b545-12df275db084" />

我做了以下事情来排查原因：
1. 固定seed；
2. 统一图片读取为RGB 
3. 修改internvl3-1b 下的config 去除template 中的system prompt
4. 同一batch_size 为 1.
均没有拉近这个差距，请问是为什么呢 

Do you set `--limit` to second evaluation method?

> Do you set `--limit` to second evaluation method?

The -limit I understand is to make a choice on the quantity of the evaluation data, isn't it? In fact, for both of my evaluation methods, I have set a -limit of 100

Yes I just to confirm the setting is correct if both are evaluating the same quantity of data.

> Yes I just to confirm the setting is correct if both are evaluating the same quantity of data.

And I also confirmed the output logs, which are 100 identical pieces of data, but their results are different, as follows:

<img width="1098" height="574" alt="Image" src="https://github.com/user-attachments/assets/e9c7f046-9a1a-4cf6-8603-d4d70efc44df" />

I was suspecting that if it's appropriate to use `internvl2` to run internvl3? 

The vllm serve method maybe more correct at least it's a standard way of hosting any model.

> I was suspecting that if it's appropriate to use `internvl2` to run internvl3?
> 
> The vllm serve method maybe more correct at least it's a standard way of hosting any model.


In fact, I also reached a similar conclusion using qwen2-vl. The results of offline and online are different. Just now I found that the maximum value of the cut image for online and offline is different, which will have a certain impact. For online, it is 12, and for offline, it is 6. After modifying online to 6, the docvqa results were similar, but there were still some differences in other datasets.


It was found that in the online mode, the chat_template under the internvl3 model file is ali, which leads to inconsistent subsequent queries and is the main reason for the inconsistent metrics.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug while initializing the evaluation of Qwen2VL-7B
Root cause: Environment Incompatibility

IndexError: index 0 is out of bounds for dimension 0 with size 0 for " if cache_position is None or (cache_position is not None and cache_position[0] == 0):

The bug is similar to https://github.com/QwenLM/Qwen2.5-VL/issues/596. Is replacing the template_chat JSON file methodologically sound? Could it potentially compromise model performance?"

Can you try `pip install -U transformers`? I think this issue is possibly related to transformers and not lmms-eval. I am using transformers==4.52.3 and seems I can not reproduce this error

Yes, thanks. This is due to the version of transformers, thanks.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Issues when using llava_vid.py to evaluate the llava-onevision-qwen2-7b-ov model
Root cause: Configuration Error

I used the **lmms-lab/llava-onevision-qwen2-7b-ov** model for evaluation by calling it in the lmms_eval/models/**llava_vid.py** file. Why are the results so poor, with **accuracy only in the teens**? From what I can see, there doesn't seem to be much difference in the evaluation process between llava_vid.py and llava_onevision.py.
I first stored information from multiple frames of the benchmark into **a single file**. The **llava_vid.py file is more suitable for this**, as it processes the questions and answers one by one. On the other hand, the llava_onevision.py file splits the questions and video into chunks, which makes it more difficult to handle.That's exactly **why I want to use the llava_vid.py file**.

The file diff should not make much impact on results. Have you checked the model responses are seemingly correct? like without repeated patterns or have correct instruction following ability?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: VideoMMMU Results with Qwen2.5-VL
Root cause: Documentation Deficiency

Thank you for your excellent work.

I’ve been trying to evaluate Qwen2.5-VL-7B on the VideoMMMU, but my current setup only achieves around 10% accuracy. I suspect there may be an issue with my evaluation script.

Could you kindly share the recommended command or configuration for evaluating Qwen2.5-VL on Video-MMMU? Specifically, should we use qwen2_5_vl or qwen2_5_vl_interleave? Also, may I ask what performance results you obtained using this model?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Dataset lmms-lab/OlympiadBench does exist on Hugging Face
Root cause: Configuration Error

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/olympiadbench/olympiadbench_test_en.yaml and https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/olympiadbench/olympiadbench_test_cn.yaml has:

```yaml
lmms-lab/OlympiadBench
```

But lmms-lab/OlympiadBench does exist on Hugging Face.

It's been deprecated and we recommended to use the `olympiadbench_testmini` for evaluating using official provided dataset.

```sh
dataset_path: Hothan/OlympiadBench
dataset_name: OE_MM_maths_en_COMP
```

Ofc you can adjust the config yaml to evaluate on more subsets of Hothan/OlympiadBench.

Do we want to delete these deprecated stuff? If yes, I can produce a PR to do it.

> Do we want to delete these deprecated stuff? If yes, I can produce a PR to do it.

Yes, sure please go ahead! And we also made modifications in branch `dev/v0d4` that incorporates the tasks from MIMO-VL.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Slow execution
Root cause: Documentation Deficiency

我在没有网的服务器中设置hf-datasets_offline=true,并且指定数据集的本地路径，在第一次执行的时候速度特别慢，在第一次执行结束有指标之后第二次执行很快，这是为什么


怎么指定数据集的本地路径呀
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Getting stuck when download some video benchmarks
Root cause: Resource Mishandling

When evaluating datasets like longvidebench, perceptiontest, videommmu and mlvu_test, the program gets stuck either at the beginning or in the middle of fetching files. I've already tried using hf-mirror, adding hf_token, and signing the dataset usage agreement on Hugging Face. Do you have any suggestions? This doesn't happen with datasets like mvbench.
Like:
Fetching 7 files:   0%|          | 0/7 [00:00<?, ?it/s]
Fetching 12 files:   0%|          | 0/12
or:
Fetching 12 files:   16%|||||      | 2/12

I think this is because of a dramatic speed drop so that the hf download would stuck for a long time before the next update. I would suggest you first fetching the huggingface file using snapshot download, then let the lmms-eval to unzip all the files
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Too many requests for MathVista
Root cause: Validation Gap

I'm trying to run evaluations for my model on the MathVista dataset. 
I received this error message:
```
[tasks/mathvista/mathvista_evals.py:86] ERROR 429 Client Error: Too Many Requests for url: https://api.openai.com/v1/chat/completions
```
But I checked my api account that I have prepaid and it worked fine. Why would it show that it was receiving too many requests?


Do you fix this? I meet the same question