Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Abnormal GPU Utilization When Evaluating Qwen 2.5 VL 72B
Root cause: Validation Gap

# Low GPU Utilization and Errors During Qwen 2.5 VL 72B Evaluation

First, I'd like to commend the lmms-eval team for developing such a robust evaluation framework - it's been incredibly valuable for multimodal research.

## Problem Description
When evaluating Qwen 2.5 VL 72B on 8x H100 80G GPUs using the following command:
```bash
accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval \
  --model qwen2_5_vl \
  --model_args="pretrained=/mnt/mypath/models/Qwen2.5-VL-72B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False,device_map=auto" \
  --tasks super_clevr \
  --output_path /mnt/mypath/qwen72b \
  --batch_size 1 \
  --verbosity DEBUG
```

My GPU utilization only reaches a maximum of 10%. 

<img width="1480" height="483" alt="Image" src="https://github.com/user-attachments/assets/c4bf93a3-a7bc-4834-8eae-ee1062b7f321" />

## Attempted Solutions and Errors
1 Increasing num_processes to 8: Results in OOM (Out of Memory) errors despite using H100 GPUs 80G

This is most likely because the code in [this](../blob/main/lmms_eval/models/qwen2_5_vl.py#L75) ignores the device_map passed in by the user when num_processes is greater than 1 and uses f `“cuda:{accelerator.local_process_index}”`, which results in the model not being sharded correctly

```python
accelerator = Accelerator()
if accelerator.num_processes > 1:
    self._device = torch.device(f"cuda:{accelerator.local_process_index}")
    self.device_map = f"cuda:{accelerator.local_process_index}"
else:
    self._device = torch.device(device)
    self.device_map = device_map if device_map else device
```

2 Increasing batch_size > 1: Throws ValueError:
```bash
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen2_5_VL. 
Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input
```

After that, I tried to change [code](../blob/main/lmms_eval/models/qwen2_5_vl.pyL102) :`self._tokenizer = AutoTokenizer.from_pretrained(pretrained)` in `qwen2_5_vl.py` to `self._tokenizer = AutoTokenizer.from_pretrained(pretrained, padding_side = 'left')` but it didn't work.

##  Request for Assistance
How should I properly configure either:
1. Multi-GPU evaluation to avoid OOM, or
2. Batched evaluation with correct padding configuration?
3. Any other recommended parameters for efficient evaluation of this large vision-language model?

We'd be grateful for any insights from the team or community members who have successfully evaluated similar large multimodal models.



I would recommend you to use vllm or sglang to evaluate the 72B models. You can either serve it or serve it as an api server and then use openai to evaluate it

I tried using `vllm`, but depending on performance with the use of `accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval  --model qwen2_5_vl` is different (4-5% lower accuracy for vllm), this is my vllm script, is there something wrong with my setup?

```bash
set -x

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_BLOCKING_WAIT=1
export NCCL_TIMEOUT=18000000
export NCCL_DEBUG=DEBUG
export TORCH_CUDA_ARCH_LIST="9.0"

python3 -m lmms_eval \
    --model vllm \
    --model_args model_version=${model_path},tensor_parallel_size=4,gpu_memory_utilization=0.9 \
    --tasks my_task \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs \
    --verbosity DEBUG
```

Because I noticed that a parameter like max_pixels=12845056 doesn't exist in the `vllm model` code implemented by `lmms-eval`. I'd appreciate it if you could guide me.

I am a bit unsure about whether it is possible to override image processor config with vllm commands directly. The most easy way is that you modify your local image processor json and then start server using that path

Hi @kcz358 
I think I know what the problem is, it's because of a bug in the vllm backend in the lmms-eval [code](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/8ada80501d0e284abc2821d3a2454e1944fc133c/lmms_eval/models/vllm.py#L33)

Specifically, the `encode_video` function in the code returns a list in the call
```python
def encode_video(self, video_path):
    vr = VideoReader(video_path, ctx=cpu(0))
    total_frame_num = len(vr)
    uniform_sampled_frames = np.linspace(0, total_frame_num - 1, self.max_frame_num, dtype=int)

    # Ensure the last frame is included
    if total_frame_num - 1 not in uniform_sampled_frames:
        uniform_sampled_frames = np.append(uniform_sampled_frames, total_frame_num - 1)

    frame_idx = uniform_sampled_frames.tolist()
    frames = vr.get_batch(frame_idx).asnumpy()

    base64_frames = []
    for frame in frames:
        img = Image.fromarray(frame)
        output_buffer = BytesIO()
        img.save(output_buffer, format="PNG")
        byte_data = output_buffer.getvalue()
        base64_str = base64.b64encode(byte_data).decode("utf-8")
        base64_frames.append(base64_str)
     return base64_frames
```

This list will be appended to `all_tasks` and eventually aggregated to `imgs`, using the following code, where the `img` in the f-string expression may be a list, not a str：

```python
.......
    imgs = []  # multiple images or frames for video
    all_tasks = []
    with ThreadPoolExecutor(max_workers=self.threads) as executor:
        for visual in visuals:
            if isinstance(visual, str) and (".mp4" in visual or ".avi" in visual or ".mov" in visual or ".flv" in visual or ".wmv" in visual):
                all_tasks.append(executor.submit(self.encode_video, visual))
.......
        for task in all_tasks:
            imgs.append(task.result()) # base64_frames list be append in imgs

messages = [{"role": "user", "content": []}]
messages[0]["content"].append({"type": "text", "text": contexts})
for img in imgs:
    messages[0]["content"].append({"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img}"}})

```
 As a result, the model receives invalid URLs such as data:image/png;base64,[‘AAA’,'BBB',...] and will not be able to get any video information.

So when I use the vllm backend instead of the qwen25vl, the model never gets it right when it encounters the video, which results in a performance drop.

I'd be happy to submit a pr to fix it if you're sure it's a bug!

Yeah I think you are correct. I saw you are PR about this. Do you mind also give me a reproduce script so I can test the fix is correct? After testing and reviewing, if the performance drop is fixed, I will merge the PR. Thanks!

Hi, I want to know what's the version of your Transformers during evaluating Qwen?

Ok, my dataset is not open source yet, but it can be reviewed on other video datasets supported on lmms-eval, such as vsi-bench
```bash
set -x

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_BLOCKING_WAIT=1
export NCCL_TIMEOUT=18000000
export NCCL_DEBUG=DEBUG

model_path=/mnt/your_models_path/Qwen2.5-VL-3B-Instruct

python3 -m lmms_eval \
    --model vllm \
    --model_args model_version=${model_path},tensor_parallel_size=1,gpu_memory_utilization=0.9 \
    --tasks vsibench \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs \
    --verbosity DEBUG
```

You can try this script

Hi @tianjiedai， this is my version of  transformers and qwen-vl-utils
<img width="854" height="102" alt="Image" src="https://github.com/user-attachments/assets/75867fcf-3769-4902-b90f-20f20452ac81" />