zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The results obtained by using vllm and vllm generate differ significantly.
Root cause: Algorithmic Error

Previously, I used the simple version of vllm to test multiple models on various test sets. Recently, I discovered the updated vllm generate and retested some models. I found significant differences on some datasets for certain models (using the same set of parameters, only the `--model` option was modified). For example, `Qwen3vl-8b-instruct` yielded scores of 59.6 and 29.4 on mmvu and scivideobench respectively using vllm generate; however, the scores using vllm generate were 57.9 and 26.2 respectively. I saw in the vllm generate comments that it handles Qwen models specially, but all the models I tested were Qwen series models or variants obtained by fine-tuning Qwen models. Yet, some models showed little difference after being changed. Is this normal?

I also tested the performance of qwen3vl-8b-thinking on videommmu. The result on vllm simple was 66.56, while the result on vllm generate was only 59.56, while the official result is around 70.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: process_results_use_image is not working
Root cause: Documentation Deficiency

It appears that the `process_results_use_image` attribute, defined [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/36393e9340cf53032d697e308cafaf2f519481bf/lmms_eval/api/task.py#L89), is unused within the codebase. Additionally, I could not find any documentation describing this parameter.

I assume this parameter is intended to enable access to image columns within the `process_results` function. Could you please clarify its purpose?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: There is a bug in OCRBench accuracy compute func.
Root cause: Algorithmic Error

<img width="710" height="753" alt="Image" src="https://github.com/user-attachments/assets/c0afe076-44ba-4d85-95a1-e9acc85794c5" />
the final score always devide 1000 when i set any limited num of data to evaluate `--limit 0.1` . the right code should be this below:

<img width="594" height="56" alt="Image" src="https://github.com/user-attachments/assets/716e27ed-927e-4f31-8073-ff432c452b71" />
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: I have a question regarding VLLM parameter settings.

I want to set the maximum number of pixels and the maximum number of frames for my video. Should I set these? `max_pixels=200704, max_frame_num=16


python -m lmms_eval \
    --model vllm \
    --model_args model=${MODEL},tensor_parallel_size=${TENSOR_PARALLEL_SIZE},data_parallel_size=${DATA_PARALLEL_SIZE},gpu_memory_utilization=${GPU_MEMORY_UTILIZATION},max_pixels=200704,max_frame_num=16,disable_mm_preprocessor_cache=True,enable_prefix_caching=False \
    --tasks ${TASKS} \
    --batch_size ${BATCH_SIZE} \
    --log_samples \
    --log_samples_suffix ${LOG_SUFFIX} \
    --use_cache ./cache/sqlite_cache.db \
    --output_path ${OUTPUT_PATH} \
    --force_simple \
    --predict_only
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: 用vllm和不用vllm结果差的很多
Root cause: Algorithmic Error

我在videomme和mvbench上测试，分别用vllm和qwen2.5-vl，得到的结果差别很大，用vllm分数要低很多：
videomme: 57.52，用vllm: 54.11
mvbench: 62.8, 用vllm: 57.725

我的结果和您的结果相差也很大。
videomme: 60.1
mvbench: 65.7
我有疑惑是不同机子相差也这么大吗？

> 我的结果和您的结果相差也很大。 videomme: 60.1 mvbench: 65.7 我有疑惑是不同机子相差也这么大吗？

您测得是哪个模型？用的是vllm还是？
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: MVBench dataset  average result
Root cause: Unimplemented Feature Gap

Can we add an average result to the MVBench dataset? Currently, the results obtained are all from small subsets of tests.

<img width="377" height="220" alt="Image" src="https://github.com/user-attachments/assets/ef061666-b1ae-4ed0-8bbd-c8485bebd957" />
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The test results for Qwen3VL show significant discrepancies compared to the official report.
Root cause: Configuration Error

I ran the scripts provided by lmms_eval directly: vllm_qwen3vl.sh or vllm_generate_qwen3vl.sh to test qwen3vl-8b-instruct and thinking, using the tasks videomme, videommmu, and mvbench. I found that the test results differed significantly from the official report, being more than 5%, or even 10%, lower. For example, for videommmu, I ran vllm_qwen3vl.sh, only modifying the model and task.  The model used is qwen3vl_8b_thinking.

<img width="1266" height="388" alt="Image" src="https://github.com/user-attachments/assets/d75dbdb1-ae41-4a7d-8a20-37206d4fe6b4" />

The official report states 72.8, and even excluding parameters like the maximum frame rate, the difference shouldn't be more than 10%. Have you, the author, encountered this problem before?

Similar issue, I evaluated the Qwen3-VL-Thinking model on VideoMME, but the results were significantly lower than the results in the official repository. QAQ

@kcz358 请问这个问题怎么解决呀。我看到有issue针对videomme对qwen3vl.py做了修改，但是针对其他数据集呢？其次不用vllm，测试发现特别慢，两张卡对于videommmu都要跑十几个小时

不知道是不是对videomme做过优化，不用vllm，我用4张卡跑videomme速度好一点，大概1-3小时（跑着跑着又预计十几个小时了，好奇怪啊），然而跑videommmu或者mvbench就非常非常慢

accelerate launch --num_processes 4 --main_process_port 12345 -m lmms_eval \
    --model qwen3_vl \
    --model_args pretrained=${MODEL},device_map="auto",max_pixels=200704,max_num_frames=16,fps=2,attn_implementation=flash_attention_2 \
    --tasks ${TASKS} \
    --batch_size 1 \
    --log_samples \
    --use_cache ./cache/sqlite_cache.db \
    --log_samples_suffix  ${LOG_SUFFIX} \
    --output_path ${OUTPUT_PATH} \
    --predict_only
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: It costs a long time to test on the DocVQA
Root cause: External Dependency Breakage

When I use Qwen3-VL-8B Instruction to evaluate on the DocVQA_val, I waited for a long time but didn't get the final test results after running the dataset. The program seems to be stuck. My command is 
```
accelerate launch --num_processes 4 -m lmms_eval \
  --model qwen3_vl \
  --model_args=pretrained=Qwen/Qwen3-VL-8B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False \
  --tasks docvqa_val \
  --batch_size  3 \
  --limit 1500 \
  --output_path ./logs/Qwen3-VL-8B-Instruct_docvqa_limit1500_eval
```

<img width="1485" height="493" alt="Image" src="https://github.com/user-attachments/assets/8bc958a2-34cc-4290-9d9b-c84b66780e0b" />


不是数据集的问题，我也是几乎同样的命令，测试qwen3vl系列的模型，奇慢无比，450个样本要几十个小时，这是很离谱的事情。用vllm会好很多，但是vllm测试出来的qwen3vl模型的结果比官方报告总是低很多

发现上周更新了qwen3vl的chat文件，使用这个速度变得好一点了，但是还是很慢

Check if the PyTorch version is 2.9. There seem to be some issues with conv3d (which Qwen3VL uses for visual embedding) in PyTorch 2.9. Try downgrading to 2.8 or an earlier version.

> Check if the PyTorch version is 2.9. There seem to be some issues with conv3d (which Qwen3VL uses for visual embedding) in PyTorch 2.9. Try downgrading to 2.8 or an earlier version.检查一下 PyTorch 版本是否为 2.9。PyTorch 2.9 中 conv3d（Qwen3VL 用于视觉嵌入）似乎存在一些问题。尝试降级到 2.8 或更早的版本。

Hello, my Torch version is 2.8, and I'm still experiencing slow testing. Also, for qwen3vl, vllm requires at least version 0.11, which corresponds to a minimum Torch version of 2.8.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The process can't stop after responding to all samples...
Root cause: Resource Mishandling

<img width="1051" height="119" alt="Image" src="https://github.com/user-attachments/assets/16dd36a7-5d3e-439f-b399-dbe35cae21e6" />

Qwen2.5-VL-7B-Instruct on Video-MME

Fixed in #936
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The experimental results of Qwen3-VL-8B-Instruct could not be reproduced
Root cause: Configuration Error

Hello, we are currently attempting to reproduce the experimental results of Qwen3-VL-8B-Instruct based on the lmms-eval framework. However, we are unable to reproduce the official reported results on both image and video benchmarks.

Our experimental settings are as follows:

- For image benchmarks:
max_pixels = 12845056, attn_implementation = flash_attention_2, interleave_visuals = False
- For video benchmarks:
max_pixels = 301056, attn_implementation = flash_attention_2, max_num_frames = 64
(For benchmarks such as MVBench, fps = 1 is used.)

same issue

You could probably find the reason in similar issue: https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/901

I changed the prompt to be consistent with the official Qwen report, but the test still couldn't be reproduced. However, this might be because I set the maximum frame rate to 16, which is 10% lower than that in the report on videomme. Even if it's only 16 frames, I don't think a 10% difference is reasonable?



---Original---
From: "Fedor ***@***.***&gt;
Date: Fri, Dec 19, 2025 16:35 PM
To: ***@***.***&gt;;
Cc: "Lin ***@***.******@***.***&gt;;
Subject: Re: [EvolvingLMMs-Lab/lmms-eval] The experimental results ofQwen3-VL-8B-Instruct could not be reproduced (Issue #935)


superAIyah left a comment (EvolvingLMMs-Lab/lmms-eval#935)
 
You could probably find the reason in similar issue: #901
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***&gt;

@Fu-Fu-Fu-Fu Have you solved the problem?I met this too while evaluating videomme.What should I do?



@FrankYang-17 Can you share the command/script you used to run the MVBench on Qwen3-VL ? 

> [@Fu-Fu-Fu-Fu](https://github.com/Fu-Fu-Fu-Fu) Have you solved the problem?I met this too while evaluating videomme.What should I do?这个问题解决了吗？我在评估 Videomme 时也遇到了这个问题。我该怎么办？

The problem has not been solved yet.

I was able to closely reproduce result of Qwen/Qwen3-VL-4B-Thinking on MVBench. I used "--model vllm_generate"  class
Also in MVBench the max token to produce was set to 16 so had to increased that as well.

Used FPS=2 with Max Num of frames as 1024

Maybe others can try to use these configs to reproduce results for Qwen3-VL
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error when import InstructBLIP
Root cause: Validation Gap

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/24dc435908d921e9f1a5706e3141b12e5d838d18/lmms_eval/models/instructblip.py#L9

It seems that the mmmu task does not have this utils_group_img file.

Hi, sorry for the inconvenience. I forgot to cherry-pick the correspond commit from internal to public repo when fixing the instructblip error. Can you try again whether you can import instructblip now?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: MMVet results are low
Root cause: Algorithmic Error

Hi, it was found that the scores obtained during the MMVet test were unusually low. The official Qwen-VL model also only had a score of less than 5. I used vllm and vllm_generate.

<img width="1830" height="54" alt="Image" src="https://github.com/user-attachments/assets/5cba7020-c721-4617-9706-3705c9cf2237" />

same as https://github.com/dvlab-research/VisionThink/issues/17#issuecomment-3685614844
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: No target answer in voicebench
Root cause: Unimplemented Feature Gap

Hi,
I tried to test Qwen3-Omni on voicebench (aplacaeval/commoneval), but there is no target answer in datasets(https://huggingface.co/datasets/lmms-lab/voicebench/tree/main).
How to fix it?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2.5-VL Expect all tensors on the same devices
Root cause: Configuration Error

Dear author,
      when I evaluate Qwen2.5-VL-3B on only one GPU and set num_processes=1, the error occurred!
"Error during evaluation: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select). Please set `--verbosity=DEBUG` to get more information."

I've searched issues for this , but all are about multiple GPUS, so could you help me solve this. ‼️I've tried "device_map=auto",but it not works
My .sh file is as follows 

export HF_HOME="~/.cache/huggingface"
export CUDA_VISIBLE_DEVICES=0

accelerate launch --num_processes=1 --main_process_port=12344 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=./datasets/Qwen2.5-VL-3B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False,device_map=auto \
    --tasks mathvision_test \
    --device cuda:0 \
    --output_path ./logs/ \
    --batch_size 4
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Any plan for supporting Lora peft loading
Root cause: Unimplemented Feature Gap

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

- [ ] A descriptive title: [xxx] XXXX
- [ ] A detailed description

Thank you for your contributions!


Yes! I think this would not be hard since LLaVA naturally support it. But for other models, should write customized loading process.

Basically, it's a model side change, `lmms-eval` only provide the interfaces for `inputs`, `outputs` and calculate scores for each task.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Issue: LongVideoBench Video-LLaVA fails with empty frame list (ValueError: need at least one array to stack)
Root cause: Validation Gap

Hi,

I’m running `lmms-eval` on LongVideoBench with the `video_llava` model and consistently hit a failure at the very start of generation:

**Error**
`ValueError: need at least one array to stack`

**Stack trace (relevant part)**

* `lmms_eval/models/video_llava.py:187` calls `read_video_pyav(visuals[0], self.num_frames)`
* `lmms_eval/models/model_utils/load_video.py:84`:

  ```python
  return np.stack([x.to_ndarray(format="rgb24") for x in frames])
  ```
* `frames` is empty → `np.stack([])` raises `ValueError: need at least one array to stack`

**Repro command**

```bash
python -m accelerate.commands.launch --num_processes=8 -m lmms_eval \
  --model video_llava \
  --tasks longvideobench_val_v \
  --batch_size 1 \
  --log_samples \
  --log_samples_suffix video_llava_lvb_v \
  --output_path ./logs/
```

**Environment**

* Python: 3.11.14 (conda env)
* accelerate: installed (launch works)
* torch: (installed; CUDA device is visible, model loads on `cuda:0`)
* transformers: present (warning about `TRANSFORMERS_CACHE` deprecated)
* Running on an HPC cluster (Northeastern Explorer), but the failure occurs after the model loads and the task builds contexts successfully.

**Observed behavior**

* Task builds contexts successfully: `1337/1337` contexts built.
* Fails immediately when entering `generate_until`, at the first `read_video_pyav` call.

**Expected behavior**

* The video loader should return at least one frame for valid videos, or gracefully handle unreadable/empty videos (e.g., retry with a different backend, skip sample, or raise a clearer error including the video path / doc_id).

**Request**
Could you please:

1. Add a guard in `read_video_pyav` to handle the case where `frames` is empty (raise a clearer exception with `video_path`, or return a placeholder), and/or
2. Add logging to print the offending `video_path` / `doc_id` / split when `frames` is empty, so users can debug dataset integrity/path issues.

If helpful, I can rerun with `--verbosity=DEBUG` and share the first failing `doc_id`/video path once I add a print around the callsite.

Thanks!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The command-line argument "batch_size" being set to “auto” is applicable to all models?
Root cause: Validation Gap

When I use lmms_eval in the following manner, the code throw exception that 'ValueError: invalid literal for int() with base 10: 'auto'.

accelerate launch $CONFIG_FILE_ARG \
    -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained="${PRETRAINED_MODEL_PATH},min_pixels=${MIN_PIXELS},max_pixels=${MAX_PIXELS},attn_implementation=flash_attention_2,interleave_visuals=False" \
    --tasks $TASKS \
    --verbosity WARNING \
    --batch_size auto \
    --log_samples \
    --log_samples_suffix $RUN_NAME \
    --output_path $LOG_DIR/$RUN_NAME
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to contribute a new dataset?
Root cause: Documentation Deficiency

We (ByteDance + Huazhong University of Science and Technology) have recently constructed a benchmark(MTVQA) to measure the multilingual text comprehension ability of LMMs, including 9 widely-used but low-resource languages, which demonstrates that there is still a large room for LMMs to improve the ability of multilingual text perception and comprehension. MTVQA bench wishes to elevate the multimodal research community's attention to a wider range of visual texts. Would it be possible to add MTVQA to your collection?  
MTVQA: https://github.com/bytedance/MTVQA

Hi, you can refer to PR such as 

#95 #28 #100 #52 

So basically you need to first prepare your dataset on huggingface and then prepare the task config in the tasks folder
Repository: Farama-Foundation/Metaworld
Issue: Tests for V2 environments
Root cause: Unimplemented Feature Gap

Right now a lot of the tests in metaworld reference `ALL_V1_ENVIRONMENTS`, `MEDIUM_MODE_ARGS_KWARGS`, `HARD_MODE_CLS_DICT`, etc. Since the V2 environments are not included in these structures, they aren't covered by our tests.

closed via #312
Repository: Farama-Foundation/Metaworld
Issue: Scripted policies for ML45/MT50 environments
Root cause: Unimplemented Feature Gap

- [x] assembly @haydenshively 
- [x] bin-picking @avnishn 
- [ ] box-close @haydenshively 
- [ ] button-press-topdown-wall @haydenshively 
- [x] button-press
- [ ] button-press-wall @haydenshively 
- [x] coffee-button @haydenshively 
- [x] coffee-pull @haydenshively 
- [x] coffee-push @haydenshively 
- [x] dial-turn
- [x] dissassemble @haydenshively 
- [x] door-lock @haydenshively 
- [x] door-unlock @haydenshively 
- [x] faucet-close @haydenshively 
- [x] faucet-open @haydenshively 
- [ ] hammer
- [x] hand-insert
- [x] handle-press-side
- [x] handle-press
- [x] handle-pull-side
- [x] handle-pull
- [x] peg-unplug-side
- [x] pick-out-of-hole
- [x] pick-place-wall
- [x] plate-slide-back-side
- [x] plate-slide-back
- [x] plate-slide-side
- [x] plate-slide
- [x] push-back
- [x] push-wall
- [x] reach-wall
- [x] soccer
- [ ] stick-pull @adibellathur 
- [ ] stick-push @adibellathur 
- [x] window-close

check list for pull requests that are up, but not yet merged:
- [x] assembly
- [x] bin-picking @avnishn 
- [x] box-close
- [x] button-press-topdown-wall
- [x] button-press
- [x] button-press-wall
- [x] coffee-button
- [x] coffee-pull
- [x] coffee-push
- [x] dial-turn
- [x] dissassemble
- [x] door-lock
- [x] door-unlock
- [x] faucet-close
- [x] faucet-open
- [x] hammer
- [x] hand-insert
- [x] handle-press-side
- [x] handle-press
- [x] handle-pull-side
- [x] handle-pull
- [x] peg-unplug-side
- [x] pick-out-of-hole
- [x] pick-place-wall
- [x] plate-slide-back-side
- [x] plate-slide-back
- [x] plate-slide-side
- [x] plate-slide
- [x] push-back
- [x] push-wall
- [x] reach-wall
- [x] soccer
- [x] stick-pull
- [x] stick-push
- [x] window-close
Repository: Farama-Foundation/Metaworld
Issue: access to reward function
Root cause: Documentation Deficiency

Can we have access to reward function of MT environments? Some methods like relabeling need to know reward function.
Thanks!

@Banmahhhh the reward functions are documented in the paper and are available in the `compute_reward` function on each environment.

I'd like to note that metaworld is not intended as a general off-policy RL benchmark and isn't intended to test settings in which you have access to a reward signal.
Repository: Farama-Foundation/Metaworld
Issue: CI times out after 50 minutes
Root cause: Configuration Error

With all the new scripted policies, tests take longer than ever and builds are failing because Travis times out after 50 minutes.

Can you get under the limit by reducing the number of trials you run per scripted policy? 1000 is likely overkill.

Yes, but with fewer trials, there's a higher chance of having an unlucky test. I'll have to decrease the success rates slightly in order to get things to pass consistently.

Yes. That is always the trade-off with statistical testing. Realistically, the difference between 100 and 1000 trials is not super large. The sampling error grows at approximately `O(1/sqrt(n))` so 100 to 1000 is not as large a jump as it seems.

If you want to keep around the 1000-trial tests for local usage, your test suite can detect that it's in the CI using the `CI` environment variable and react accordingly, or you can create separate tests with more trials, mark them as `huge` (or something else), and then exclude that marker when running in the CI. I think approach #2 is preferable and you can see it used throughout garage.
Repository: Farama-Foundation/Metaworld
Issue: Some Scripted Policies are Flaky
Root cause: Validation Gap

While manually running the tests, I noticed that some environments have some variance in their average success rate, especially with running scripted policies with gaussian noise.

The variance is never high ~1-5 percent. I suggest that we allow for a tolerance of 5 percent, in order to avoid flaky tests.

Simply lowering the success rate until it's not flaky is a much clearer way of specifying the test. It tells you the minimum success rate given variance.

Will do. Instead of lowering by 5%, we'll lower by by 1% increments

We haven't had any failures recently as far as I know, so I think it's safe to close this
Repository: Farama-Foundation/Metaworld
Issue: SAC results source code
Root cause: Documentation Deficiency

Hello, the meta-world paper is reporting results using SAC and PPO. Unfortunately, we have hard-time reproducing the SAC (and PPO) results presented in the various settings, would it be possible to provide the SAC code of the paper in this repository or maybe pointing out to the implementation of SAC which was used? Thanks!

Hi @perezjln, thanks for checking out Metaworld!

The implementation of SAC that was used is available in the [rail soft-learning repository](https://github.com/rail-berkeley/softlearning). Unfortunately, I don't know the implementation of PPO that was used, as I was not an author on metaworld. The implementations of MTSAC and MTPPO that were used in the Metaworld paper are not publicly available. 

However, 

Some of the maintainers, including myself, have made implementations of the algorithms that you're looking for under a repository called [Garage](https://github.com/rlworkgroup/garage). Garage has stable implementations of [SAC](https://github.com/rlworkgroup/garage/blob/master/examples/torch/sac_half_cheetah_batch.py), [PPO](https://github.com/rlworkgroup/garage/blob/master/examples/torch/ppo_pendulum.py), [MTSAC](https://github.com/rlworkgroup/garage/blob/master/examples/torch/mtsac_metaworld_mt10.py), and [MTPPO](https://github.com/rlworkgroup/garage/blob/master/examples/torch/mtppo_metaworld_mt10.py).

I hope this answers your question; since this isn't a direct issue with Metaworld, I'm going to close this for now, but feel free to ask more questions if you have any!

PPO results used garage MTPPO (in TensorFlow).