Repository: open-compass/VLMEvalKit
Issue: Would VLLM be used to accelerate model inference?
Root cause: Unimplemented Feature Gap

I check the repository code and noticed that the current inference seems to still be limited to single sample each GPU. Would it be possible to support VLLM to achieve batch inference acceleration?

same question 

Hi,   

We now support [LMDeploy](https://github.com/InternLM/lmdeploy), a powerful tool designed to accelerate the inference of LLMs and MLLMs, similar to **VLLM** .   

If you’re planning to use VLLM instead, the execution code is quite similar. You can simply adapt the existing implementation to align with VLLM's requirements. If you need guidance on making the necessary modifications, feel free to ask—we’re happy to assist!   

@PhoenixZ810 If I understand correctly, you use LMDeploy to serve local models as APIs, and then evaluate the API. So you are suggesting that vLLM can do the same thing. I think we are curious about if the vLLM or LMDeploy can be directly implemented in the model part instead of using it as an API server? If it is already implemented somewhere, can you kindly point me to it? Or can you share some thoughts about how to implement it?

@PhoenixZ810 can lmdeploy use multiple gpus? right now, it is extremely slow to eval, especially when evaluating r1-like models with tens of thousands of output tokens

@ligeng0197 You can use [vLLM](https://github.com/vllm-project/vllm) or [LMDeploy](https://github.com/InternLM/lmdeploy) as a server, and config the `VLMEvalKit` client with `--model lmdeploy` like:

**vLLM server:**

```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
    --max-model-len=16384 \
    --limit-mm-per-prompt image=3,video=2 \
    --gpu-memory-utilization 0.9 \
    --port 23333 \
    -tp 1
```

**VLMEvalKit client:**

```bash
python run.py \
        --data MMBench_DEV_EN \
        --model lmdeploy \
        --verbose \
        --api-nproc 16
```

@YushunXiang Hello! Does this vLLM server support multi-card inference?

> [@YushunXiang](https://github.com/YushunXiang) Hello! Does this vLLM server support multi-card inference?

Of course.

@YushunXiang Hi! Thank you for your reply! Could I use the config file of the VLMEvalKit when I use the lmdeploy or vllm? How to use the config in these commends?

@YushunXiang 
How to set the configuration? For example I want to run inference with the config.json like
```
{
    "model": {
        "Qwen2-VL-2B-Instruct": {
            "class": "Qwen2VLChat",
            "model_path": "Qwen/Qwen2-VL-2B-Instruct",
            "max_pixels": 6272000,
        }
    },
    "data": {
        "MMBench_DEV_EN_V11": {
            "class": "ImageMCQDataset",
            "dataset": "MMBench_DEV_EN_V11"
        }
    }
}
```

What's more, in this vLLM server setting, will the customized prompt of "Qwen2VLChat" model (in `vlmeval.vlm.qwen2_vl.prompt.Qwen2VLPromptMixin.build_prompt`)take effect?

Could you provide an example? 

Thanks a lot!

> [@YushunXiang](https://github.com/YushunXiang) How to set the configuration? For example I want to run inference with the config.json like
> 
> ```
> {
>     "model": {
>         "Qwen2-VL-2B-Instruct": {
>             "class": "Qwen2VLChat",
>             "model_path": "Qwen/Qwen2-VL-2B-Instruct",
>             "max_pixels": 6272000,
>         }
>     },
>     "data": {
>         "MMBench_DEV_EN_V11": {
>             "class": "ImageMCQDataset",
>             "dataset": "MMBench_DEV_EN_V11"
>         }
>     }
> }
> ```
> 
> What's more, in this vLLM server setting, will the customized prompt of "Qwen2VLChat" model (in `vlmeval.vlm.qwen2_vl.prompt.Qwen2VLPromptMixin.build_prompt`)take effect?
> 
> Could you provide an example?
> 
> Thanks a lot!

I am so sorry that I have not used a config file to test this, but I think the following config might be right:

```json
{
    "model": {
        "lmdeploy": {
            "class": "LMDeployAPI"
        }
    },
    "data": {
        "MMBench_DEV_EN_V11": {
            "class": "ImageMCQDataset",
            "dataset": "MMBench_DEV_EN_V11"
        }
    }
}
```

You can test it by yourself.

@YushunXiang Thanks! But how to config the "max_pixels" for qwen?

Does this require implement generate_inner in the clent model by ourselfs?

> [@ligeng0197](https://github.com/ligeng0197) You can use [vLLM](https://github.com/vllm-project/vllm) or [LMDeploy](https://github.com/InternLM/lmdeploy) as a server, and config the `VLMEvalKit` client with `--model lmdeploy` like:您可以使用 [vLLM](https://github.com/vllm-project/vllm) 或 [LMDeploy](https://github.com/InternLM/lmdeploy) 作为服务器，并使用 `--model lmdeploy` 配置 `VLMEvalKit` 客户端，如下所示：
> 
> **vLLM server:  vLLM 服务器：**
> 
> vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
>     --max-model-len=16384 \
>     --limit-mm-per-prompt image=3,video=2 \
>     --gpu-memory-utilization 0.9 \
>     --port 23333 \
>     -tp 1
> **VLMEvalKit client:  VLMEvalKit 客户端：**
> 
> python run.py \
>         --data MMBench_DEV_EN \
>         --model lmdeploy \
>         --verbose \
>         --api-nproc 16

请问如果想要指定ip和port的话，有类似的方案吗

> lmdeploy

```
export LMDEPLOY_API_BASE=http://10.160.24.243:8000/v1/chat/completions

python run.py \
  --data MMBench_Video_8frame_pack \
  --model lmdeploy \
  --verbose \
  --mode all \
  --api-nproc 16
```
上面的方法可以用，谢谢！