Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2.5 VL-32B will get stuck when using vllm evaluation in lmms-eval
Root cause: External Dependency Breakage

When lmms-eval uses vllm to evaluate the effect of the Qwen2.5 VL-32B model on mmmu_val, it will get stuck during the vllm deployment stage. What is the reason for this? My code configuration is as follows：

<img width="1664" height="694" alt="Image" src="https://github.com/user-attachments/assets/dbff711b-1da4-4958-be25-e47444f58f01" />

<img width="2782" height="1388" alt="Image" src="https://github.com/user-attachments/assets/f24666cc-8fd1-4798-81c6-ac55285b9a3a" />


hi, did you solve it? I meet the same issue.

I think this might related to the launcher method in some settings in vllm. Not sure it this can be solved by launching with accelerate or torchrun. I will try to look into this recently

Hi, I run this command

```bash
accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval \
    --model vllm \
    --model_args model=Qwen/Qwen2.5-VL-32B-Instruct,tensor_parallel_size=4,enforce_eager=True \
    --tasks mmmu_val \
    --batch_size 1024 \
    --log_samples \
    --output_path ./logs --verbosity DEBUG --num_fewshot 0 --limit 32
```
and this seems to work for me on the newest main. Can you try this and see if it works?

@kcz358 I met the same issue with Qwen2.5-VL-32B. I tried your command with the latest main but the vllm still hangs. May I know your env info? e.g. torch, transformers and vllm version.

I am using vllm==0.11.0 and transformers==4.57.1 . A too old vllm might not having this feature to launch with external launcher

Thanks for the quick reply. I run the commands you mention:
```
accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval \
    --model vllm \
    --model_args model=/data/xxx/ckpt/Qwen2.5-VL-32B-Instruct,tensor_parallel_size=4,enforce_eager=True \
    --tasks mmmu_val \
    --batch_size 1024 \
    --log_samples \
    --output_path ./logs --verbosity DEBUG --num_fewshot 0 --limit 32
```
with the latest version lmms-eval, torch==2.8.0, vllm==0.11.0 and transformers==4.57.1 (on H800), but the code still hangs. The following is the printed message, any idea?

```
...
(EngineCore_DP0 pid=1031989) INFO 11-27 16:36:31 [core.py:644] Waiting for init message from front-end.                                                                                                                                                                          
(EngineCore_DP0 pid=1031989) INFO 11-27 16:36:31 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='/data/xxx/ckpt/Qwen2.5-VL-32B-Instruct', speculative_config=None, tokenizer='/data/xxx/ckpt/Qwen2.5-VL-32B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=128000, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=1, served_model_name=/data/xxx/ckpt/Qwen2.5-VL-32B-Instruct, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":null,"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":0,"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":0,"local_cache_dir":null}                                                  
(EngineCore_DP0 pid=1031989) INFO 11-27 16:36:31 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 16777216, 10, 'psm_f2579478'), local_subscribe_addr='ipc:///tmp/e89fdf6c-f744-434b-98eb-ba2bf5e083f7', remote_subscribe_addr=None, remote_addr_ipv6=False) 
INFO 11-27 16:36:36 [__init__.py:216] Automatically detected platform cuda.               
INFO 11-27 16:36:37 [__init__.py:216] Automatically detected platform cuda.
INFO 11-27 16:36:37 [__init__.py:216] Automatically detected platform cuda.
INFO 11-27 16:36:37 [__init__.py:216] Automatically detected platform cuda.
INFO 11-27 16:36:51 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_c2568c4f'), local_subscribe_addr='ipc:///tmp/9952f698-bfa4-400d-97bd-b7729373a8e0', remote_subscribe_addr=None, remote_addr_ipv6=False)                                                                                                                          
INFO 11-27 16:36:51 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_cdba7142'), local_subscribe_addr='ipc:///tmp/b47f2c7e-147e-43e9-828e-28b59010e860', remote_subscribe_addr=None, remote_addr_ipv6=False)                                                                                                                          
INFO 11-27 16:36:51 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_ea1aae2c'), local_subscribe_addr='ipc:///tmp/e1a68a38-97a7-4153-8fbc-c992ec161581', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 11-27 16:36:51 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_9ba6de42'), local_subscribe_addr='ipc:///tmp/83b861a2-ea9e-467e-abe3-52e07fd6890c', remote_subscribe_addr=None, remote_addr_ipv6=False)
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
INFO 11-27 16:36:53 [__init__.py:1384] Found nccl from library libnccl.so.2
INFO 11-27 16:36:53 [__init__.py:1384] Found nccl from library libnccl.so.2
INFO 11-27 16:36:53 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 11-27 16:36:53 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 11-27 16:36:53 [__init__.py:1384] Found nccl from library libnccl.so.2
INFO 11-27 16:36:53 [pynccl.py:103] vLLM is using nccl==2.27.3
INFO 11-27 16:36:54 [__init__.py:1384] Found nccl from library libnccl.so.2
INFO 11-27 16:36:54 [pynccl.py:103] vLLM is using nccl==2.27.3

```

<img width="1914" height="708" alt="Image" src="https://github.com/user-attachments/assets/3ffddc3d-f4d2-4ff5-a9c0-0ff1dcb0a2d4" />

Hi, I tried to reproduce this using your command but it seems to work on my machine. Since you are hanging after the logs prompting nccl, maybe you can try to set env variables such as nccl debug info to see more logs? Also can try if you can set the log info of vllm to debug to see more info.