Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2.5-VL-7B Charades-STA results significantly lower than reported (29.46 vs 43.6 mIoU)
Root cause: Configuration Error

## Issue Description

Following some issues in the Qwen2.5-VL official repo, 
- https://github.com/QwenLM/Qwen3-VL/issues/925#issuecomment-2716599307
- https://github.com/QwenLM/Qwen3-VL/issues/837
I've tried the default setting for running results for Qwen2.5-VL-7B on the Charades-STA benchmark.
However, I'm getting significantly lower results than reported in the paper.

## Expected Results
- **Paper reports:** 43.6 mIoU

## Actual Results
- **My evaluation:** 29.46 mIoU 
- **With fps 2:** 29.78 mIoU

This is a substantial gap of ~14 mIoU.

## Setup Details
- **Model:** Qwen2.5-VL-7B (labeled as 'temporal_grounding_charades' in lmms-eval)
- **Benchmark:** Charades-STA
- **Evaluation tool:** lmms-eval with default settings
- **script (same as default):**
```
accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False \
    --tasks temporal_grounding_charades \
    --batch_size 1 \
```

## Questions
1. Is there any implementation details that I've been missing?

Wonder if these issues are related 

https://github.com/huggingface/transformers/issues/40136
https://github.com/huggingface/transformers/issues/41180

> ## Issue Description
> Following some issues in the Qwen2.5-VL official repo,
> 
> * [Experimental Details on Videos QwenLM/Qwen3-VL#925 (comment)](https://github.com/QwenLM/Qwen3-VL/issues/925#issuecomment-2716599307)
> * [Prompts and evaluation process for Video Grounding Tasks on the technical report QwenLM/Qwen3-VL#837](https://github.com/QwenLM/Qwen3-VL/issues/837)
>   I've tried the default setting for running results for Qwen2.5-VL-7B on the Charades-STA benchmark.
>   However, I'm getting significantly lower results than reported in the paper.
> 
> ## Expected Results
> * **Paper reports:** 43.6 mIoU
> 
> ## Actual Results
> * **My evaluation:** 29.46 mIoU
> * **With fps 2:** 29.78 mIoU
> 
> This is a substantial gap of ~14 mIoU.
> 
> ## Setup Details
> * **Model:** Qwen2.5-VL-7B (labeled as 'temporal_grounding_charades' in lmms-eval)
> * **Benchmark:** Charades-STA
> * **Evaluation tool:** lmms-eval with default settings
> * **script (same as default):**
> 
> ```
> accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval \
>     --model qwen2_5_vl \
>     --model_args=pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False \
>     --tasks temporal_grounding_charades \
>     --batch_size 1 \
> ```
> 
> ## Questions
> 1. Is there any implementation details that I've been missing?

My results are exactly the same as yours. Is this resolved?

I tested using vllm with fps=1 and max_pixels=151200, I am getting result with

| IoU@0.3 | IoU@0.5 | IoU@0.7 |  mIoU |
|:-------:|:-------:|:-------:|:-----:|
|   68.09 |   48.47 |   25.11 | 45.27 |

> I tested using vllm with fps=1 and max_pixels=151200, I am getting result with
> 
> IoU@0.3	IoU@0.5	IoU@0.7	mIoU
> 68.09	48.47	25.11	45.27

Thanks for your prompt reply!
I just use the default script in qwen25vl.sh without vllm:
```
accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=/weights/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False \
    --tasks temporal_grounding_charades \
    --batch_size 1 --verbosity=DEBUG --output_path ./charades_sta/jsonl
```
It appears the only difference between us is ‘max_pixels,fps’ and ‘vllm’.
I'll test your settings. Could you also share your key env? (To avoid any impact from packages like transformers)

> 
> Thanks for your prompt reply! I just use the default script in qwen25vl.sh without vllm:
> 
> ```
> accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval \
>     --model qwen2_5_vl \
>     --model_args=/weights/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False \
>     --tasks temporal_grounding_charades \
>     --batch_size 1 --verbosity=DEBUG --output_path ./charades_sta/jsonl
> ```
> 
> It appears the only difference between us is ‘max_pixels,fps’ and ‘vllm’. I'll test your settings. Could you also share your key env? (To avoid any impact from packages like transformers)

Unfortunately, 'max_pixels=151200, fps=1' doesn't work.

IOU 0.3: 44.48924731182796
IOU 0.5: 30.43010752688172
IOU 0.7: 15.241935483870968
mIOU 30.174439369676605

I am using vllm with `model=Qwen/Qwen2.5-VL-7B-Instruct,tensor_parallel_size=1,data_parallel_size=8,max_pixels=151200,fps=1,disable_log_stats=True,max_model_len=180000,hf_overrides={"max_position_embeddings":180000},limit_mm_per_prompt={"image":1000},mm_processor_cache_gb=50,max_frame_num=512` as model args. Here are the result I achieve for the submission file eval by the eval_tvg.py in the task folder

<img width="271" height="71" alt="Image" src="https://github.com/user-attachments/assets/e58b133f-fee5-48bf-a34d-1b07b0bb5636" />

And here are the logs I have. Hope this help

[inference_results_temporal_grounding_charades_2025-10-30-21-30-11.json](https://github.com/user-attachments/files/23619000/inference_results_temporal_grounding_charades_2025-10-30-21-30-11.json)

I am using transformers==4.57.1 and vllm==0.11.0