Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Bugs Collection] Performance Gap for Qwen 2.5 VL 32B when using vLLM and Qwen model file in lmms-eval
Root cause: Validation Gap

## Problem
In `lmmd-eval`, vLLM supported models can be tested with the vLLM model class. So for Qwen 2.5 VL, we can either use [lmms_eval/models/qwen2_5_vl.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/qwen2_5_vl.py) or [lmms_eval/models/vllm.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/vllm.py).

It is observed that evaluated model performance is quite different between these 2 approaches. 


## Example
```bash
#TASK_NAME=<task>
#MODEL_NAME=<name>
#NUM_PROCESSES=<number of GPUs>

python3 -m accelerate.commands.launch \
  --num_processes=$NUM_PROCESSES \
  --mixed_precision=fp16 \
  --main_process_port=29502 \
  -m lmms_eval \
  --model qwen2_5_vl \
  --model_args "pretrained=Qwen/Qwen2.5-VL-32B-Instruct,device_map=auto,use_flash_attention_2=True" \
  --tasks ${TASK_NAME} \
  --batch_size 10 \
  --limit 1000 \
  --log_samples \
  --output_path logs/${MODEL_NAME} \
  --verbosity=DEBUG
```

```python
#TASK_NAME=<task>
#MODEL_NAME=<name>
#NUM_PROCESSES=<number of GPUs>

python3 -m lmms_eval \
  --model vllm \
  --model_args "model_version=Qwen/Qwen2.5-VL-32B-Instruct,\
gpu_memory_utilization=0.9,\
tensor_parallel_size=${NUM_PROCESSES},\
dtype=bfloat16" \
  --tasks ${TASK_NAME} \
  --batch_size 10 \
  --limit 1000 \
  --log_samples \
  --output_path logs/${MODEL_NAME} \
  --verbosity=DEBUG
```

## Updates/Fix/Conclusion 🔥
- Turns out I was using the Qwen2.5VL model file from lmms-eval v0.3, where there are bugs
- lmms-eval 0.4 is working as expected

Can you try to set the `--batch_size=1` when using `qwen2_5_vl` and to see whether the results match?

I think I also found the issue that using `vllm` for Qwen series model would cause performance drop. 

I am looking into this. 

But weirdly the performance could match (`vllm` vs `qwen2_5_vl`) with our upgraded version, if you are looking for faster model inference, you could switch to `dev/v0d4` branch to use the updated repo (I am currently checking if all details could match, most of the results could as far as I know now). 

I run some object detection tasks with batch size 1 for both `vllm` and `qwen2_5_vl`. Using `python3 -m lmms_eval  --model vllm` is consistently better than the other.

## [Solved] Inconsistent results for Qwen 2.5 VL 7B on [MathVision](https://mathllm.github.io/mathvision/) 

### Metrics
mathvision reported metric: 25.1
lmms-eval `vllm`:  26.4
lmm-eval (v0.3) `qwen2_5_vl `[w Bug]: 1.1
lmm-eval (v0.4) `qwen2_5_vl `:  25.2 🔥


### Env
lmms-eval: 0.4.0.dev0 (the dev/v0d4 branch)
transformers: 4.54.0
vllm: 0.10.0


### Command

```bash
TASK_NAME="mathvision_test"
MODEL_NAME="Qwen2.5-VL-7B-Instruct"
NUM_PROCESSES=2

python3 -m accelerate.commands.launch \
  --num_processes=$NUM_PROCESSES \
  --mixed_precision=fp16 \
  --main_process_port=29502 \
  -m lmms_eval \
  --model qwen2_5_vl \
  --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,device_map=auto" \
  --tasks ${TASK_NAME} \
  --batch_size 10 \
  --limit 1000 \
  --log_samples \
  --output_path logs/${MODEL_NAME} \
```

```bash
TASK_NAME="mathvision_test"
MODEL_NAME="Qwen2.5-VL-7B-Instruct-vllm"
NUM_PROCESSES=2

python3 -m lmms_eval \
  --model vllm \
  --model_args "model_version=Qwen/Qwen2.5-VL-7B-Instruct,\
gpu_memory_utilization=0.9,\
tensor_parallel_size=${NUM_PROCESSES},\
dtype=bfloat16" \
  --tasks ${TASK_NAME} \
  --batch_size 10 \
  --limit 1000 \
  --log_samples \
  --output_path logs/${MODEL_NAME} \
```

Did you compare the outputs? 

I would suggest try again and setting the --batch_size=1 for `qwen2_5_vl` and making sure you are using the `0.4`

I observed this command maybe not fully correct: 

`--model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,device_map=auto,use_flash_attention_2=True" \`

since you would change to `attn_implementation=flash_attention_2`

I noticed that the `qwen2_5_vl` generation process takes longer than expected, maybe because of generation has repeating behaviors.

> Did you compare the outputs?
> 
> I would suggest try again and setting the --batch_size=1 for `qwen2_5_vl` and making sure you are using the `0.4`
> 
> I observed this command maybe not fully correct:
> 
> `--model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,device_map=auto,use_flash_attention_2=True" \`
> 
> since you would change to `attn_implementation=flash_attention_2`

You mean sth wrong with flash attention 2? 

In the Qwen 2.5 VL class, these are acceptable:
```python
# Validate attention implementation
valid_attn_implementations = [None, "flash_attention_2", "sdpa", "eager"]
if attn_implementation not in valid_attn_implementations:
    raise ValueError(f"attn_implementation must be one of {valid_attn_implementations}, got {attn_implementation}")
```

I mean the args `attn_implementation=flash_attention_2` should match to make sure you are using the correct branch.

## Updates/Fix/Conclusion 🔥
- Turns out I was using the Qwen2.5VL model file from lmms-eval v0.3, where there are bugs
- lmms-eval 0.4 is working as expected