Repository: open-compass/VLMEvalKit
Issue: 使用VLLM进行推理会卡死
Root cause: Configuration Error

我在以下两个地方设置了use_vllm
分别是：
config.py:
```python
    "xx": partial(
        Qwen2VLChat,
        model_path="xx",
        min_pixels=256 * 28 * 28,
        max_pixels=8192*28*28, #16384 * 28 * 28,
        # use_custom_prompt=False
        system_prompt=r"You are a helpful multimodal assistant. You are required to answer the question based on the image provided. Put your final answer in \\boxed{}.",
        use_vllm=True
    ),
```
以及执行的脚本：
```shell
# 执行评估
torchrun \
    --master_port=29600 \
    --nproc-per-node=1 \
    run.py \
    --data ${DATASETS} \
    --model ${MODEL_NAME} \
    --judge xx \
    --work-dir ${WORK_DIR} \
    --use-vllm \
    --mode infer

```

LLM创建时卡死在 从
self.llm = LLM 跳转到
LLMEngine.from_engine_args() 跳转到
v1.LLMEngine.__init__ 跳转到
v1.EngineCoreClient.make_client(), 其中执行
        if multiprocess_mode and not asyncio_mode:
            print(f'{Fore.BLUE}@v1.make_client return SyncMPClient(vllm_config({Fore.RESET}')
            return SyncMPClient(vllm_config, executor_class, log_stats)
这句语句时卡死了，请问有什么解决办法吗

Hi! From my understanding, the hang seems to happen inside EngineCoreClient.make_client(). It looks like vLLM ends up using the synchronous multiprocessing client (SyncMPClient), which can deadlock when Python is using the default fork start method on Linux. This tends to show up more easily when running under torchrun, or in distributed or multi-threaded environments.

As far as I know, this is a known class of issues in vLLM: if the process is forked after CUDA, NCCL, or logging threads have already been initialized, the child process can get stuck forever during engine startup.

A few things that might help, based on my experiments and similar reports:

- Force Python multiprocessing to use spawn (this worked best for me).
Adding the following at the very top of the entrypoint, before importing vLLM or torch, often avoids the issue:
```
import multiprocessing as mp
mp.set_start_method("spawn", force=True)
```

This avoids the fork-with-threads pattern and usually makes the hang go away.

- Avoid the synchronous multiprocess client if possible.
The hang seems to occur on the multiprocess_mode and not asyncio_mode code path. If the wrapper supports an async engine mode, enabling that should bypass SyncMPClient.

- Try a newer version of vLLM.
Some engine initialization issues have been improved in recent releases, although from what I’ve seen, fork-related deadlocks can still happen in distributed setups.
Hope this can help! I'm looking forward to your reply! 

> Hi! From my understanding, the hang seems to happen inside EngineCoreClient.make_client(). It looks like vLLM ends up using the synchronous multiprocessing client (SyncMPClient), which can deadlock when Python is using the default fork start method on Linux. This tends to show up more easily when running under torchrun, or in distributed or multi-threaded environments.
> 
> As far as I know, this is a known class of issues in vLLM: if the process is forked after CUDA, NCCL, or logging threads have already been initialized, the child process can get stuck forever during engine startup.
> 
> A few things that might help, based on my experiments and similar reports:
> 
> * Force Python multiprocessing to use spawn (this worked best for me).
>   Adding the following at the very top of the entrypoint, before importing vLLM or torch, often avoids the issue:
> 
> ```
> import multiprocessing as mp
> mp.set_start_method("spawn", force=True)
> ```
> 
> This avoids the fork-with-threads pattern and usually makes the hang go away.
> 
> * Avoid the synchronous multiprocess client if possible.
>   The hang seems to occur on the multiprocess_mode and not asyncio_mode code path. If the wrapper supports an async engine mode, enabling that should bypass SyncMPClient.
> * Try a newer version of vLLM.
>   Some engine initialization issues have been improved in recent releases, although from what I’ve seen, fork-related deadlocks can still happen in distributed setups.
>   Hope this can help! I'm looking forward to your reply!

非常感谢您的耐心回复，我尝试了您所说的办法，不过似乎没有起效。
不过现在问题已经解决了，
核心问题是：LLM创建时在torch.distributed.init_process_group卡死 
解决办法：
使用torchrun或者python3 -m torch.distributed.run运行程序时，所附带的参数master_port需要和内部的torch.distributed.init_process_group()的参数一致

如：
```shell
python3 -m torch.distributed.run \
    --master_port=$MASTER_PORT \
    run.py \
    ...
```
```python
torch.distributed.init_process_group(backend="nccl", 
    init_method=f'tcp://{os.environ.get("MASTER_ADDR", 'localhost')}:{os.environ.get("MASTER_PORT", '23456')}', 
    rank=0, world_size=1)
```
另外为了避免在运行中出现问题可以把torch.distributed.init_process_group移到run.py的开头