Repository: pytorch
Issue: CUDA initialization error with vLLM 0.5.4 and PyTorch 2.4.0+cu121
Labels: oncall: distributed

### 🐛 Describe the bug

CUDA initialization error in forked subprocesses when using **vLLM 0.5.4 with PyTorch 2.4.0+cu121**. The same code works with vLLM 0.5.0 and PyTorch 2.3.0+cu121, but fails with newer versions (vLLM 0.6.2 with PyTorch 2.5.1+cu121).

**Error Message:**
```
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```

**Environment:**
- vLLM: 0.5.4 (Also tested with 0.6.2)
- PyTorch: 2.4.0+cu121 (Also tested with 2.5.1+cu121)
- CUDA: 12.2
- GPU Driver: 535.54.03
- OS: [Add your operating system here]

**Steps to Reproduce:**
1. Install vLLM 0.5.4 and PyTorch 2.4.0+cu121
2. Set the following environment variables:
   ```bash
   export PYTHONMULTIPROCESSING_START_METHOD=spawn
   export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"  
   export VLLM_USE_SPAWN=1
   export PYTHONPATH="${PYTHONPATH}:/mnt/data/taoshuchang.tsc/IR_RAG/IRM"
   export MASTER_PORT=29500
   export MASTER_ADDR=localhost
   export WORLD_SIZE=$(echo $CUDA_VISIBLE_DEVICES | tr ',' '\n' | wc -l)
   export CUDA_DEVICE_ORDER=PCI_BUS_ID
   ```
3. Run a script that uses vLLM to load a large language model (e.g., LLaMA 3.3-70B)
4. Observe the CUDA initialization error in forked subprocesses

**Additional Context:**
- This issue doesn't occur with vLLM 0.5.0 and PyTorch 2.3.0+cu121
- vLLM 0.5.4 is required to load LLaMA 3.3, which necessitates PyTorch 2.4.0+cu121

**Questions:**
1. Is this a known issue with PyTorch 2.4.0+cu121 and newer versions?
2. Are there any workarounds or configurations to resolve this issue?
3. Is there a compatibility matrix for vLLM, PyTorch, and CUDA versions?

**Attempted Solutions:**
- Tried vLLM 0.6.2 with PyTorch 2.5.1+cu121, but encountered the same error
- Set `PYTHONMULTIPROCESSING_START_METHOD=spawn` as suggested in error message

**Full error trace:**
```
INFO 01-19 03:55:21 config.py:899] Defaulting to use mp for distributed inference
INFO 01-19 03:55:21 llm_engine.py:226] Initializing an LLM engine (v0.6.1.dev238+ge2c6e0a82) with config: model='/mnt/data/taoshuchang.tsc/IR_RAG/ckpt/hotpot_contriever/analyze_merge//hotpot_1doc_other_Meta-Llama-3-70B-Instruct_lr1e5', speculative_config=None, tokenizer='/mnt/data/taoshuchang.tsc/IR_RAG/ckpt/hotpot_contriever/analyze_merge//hotpot_1doc_other_Meta-Llama-3-70B-Instruct_lr1e5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/mnt/data/taoshuchang.tsc/IR_RAG/ckpt/hotpot_contriever/analyze_merge//hotpot_1doc_other_Meta-Llama-3-70B-Instruct_lr1e5, use_v2_block_manager=False, num_scheduler_steps=1, multi_step_stream_outputs=False, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 01-19 03:55:22 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 32 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 01-19 03:55:22 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
[1;36m(VllmWorkerProcess pid=305)[0;0m INFO 01-19 03:55:24 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 226, in _run_worker_process
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     output = executor(*args, **kwargs)
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]              ^^^^^^^^^^^^^^^^^^^^^^^^^
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 166, in init_device
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch.cuda.set_device(self.device)
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 478, in set_device
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch._C._cuda_setDevice(device)
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     raise RuntimeError(
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
[1;36m(VllmWorkerProcess pid=305)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] 
[1;36m(VllmWorkerProcess pid=307)[0;0m INFO 01-19 03:55:24 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=308)[0;0m INFO 01-19 03:55:24 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=306)[0;0m INFO 01-19 03:55:24 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=303)[0;0m INFO 01-19 03:55:24 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=304)[0;0m INFO 01-19 03:55:24 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 226, in _run_worker_process
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     output = executor(*args, **kwargs)
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]              ^^^^^^^^^^^^^^^^^^^^^^^^^
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 166, in init_device
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch.cuda.set_device(self.device)
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 478, in set_device
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch._C._cuda_setDevice(device)
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     raise RuntimeError(
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
[1;36m(VllmWorkerProcess pid=307)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] 
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 226, in _run_worker_process
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     output = executor(*args, **kwargs)
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]              ^^^^^^^^^^^^^^^^^^^^^^^^^
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 166, in init_device
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch.cuda.set_device(self.device)
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 478, in set_device
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch._C._cuda_setDevice(device)
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     raise RuntimeError(
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
[1;36m(VllmWorkerProcess pid=308)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] 
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 226, in _run_worker_process
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 226, in _run_worker_process
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     output = executor(*args, **kwargs)
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     output = executor(*args, **kwargs)
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]              ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]              ^^^^^^^^^^^^^^^^^^^^^^^^^
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 166, in init_device
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 166, in init_device
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch.cuda.set_device(self.device)
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch.cuda.set_device(self.device)
[1;36m(VllmWorkerProcess pid=303)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 478, in set_device
[1;36m(VllmWorkerProcess pid=306)[0;0m [1;36m(VllmWorkerProcess pid=303)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 478, in set_device
[1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch._C._cuda_setDevice(device)
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch._C._cuda_setDevice(device)
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
[1;36m(VllmWorkerProcess pid=303)[0;0m [1;36m(VllmWorkerProcess pid=306)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     raise RuntimeError(
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     raise RuntimeError(
[1;36m(VllmWorkerProcess pid=306)[0;0m [1;36m(VllmWorkerProcess pid=303)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
[1;36m(VllmWorkerProcess pid=306)[0;0m [1;36m(VllmWorkerProcess pid=303)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] 
ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] 
[1;36m(VllmWorkerProcess pid=302)[0;0m INFO 01-19 03:55:24 multiproc_worker_utils.py:218] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 226, in _run_worker_process
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     output = executor(*args, **kwargs)
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]              ^^^^^^^^^^^^^^^^^^^^^^^^^
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 166, in init_device
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch.cuda.set_device(self.device)
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 478, in set_device
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch._C._cuda_setDevice(device)
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     raise RuntimeError(
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
[1;36m(VllmWorkerProcess pid=304)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] 
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 226, in _run_worker_process
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     output = executor(*args, **kwargs)
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]              ^^^^^^^^^^^^^^^^^^^^^^^^^
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 166, in init_device
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch.cuda.set_device(self.device)
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 478, in set_device
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     torch._C._cuda_setDevice(device)
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]   File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233]     raise RuntimeError(
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
[1;36m(VllmWorkerProcess pid=302)[0;0m ERROR 01-19 03:55:24 multiproc_worker_utils.py:233] 
ERROR 01-19 04:05:25 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 305 died, exit code: -15
INFO 01-19 04:05:25 multiproc_worker_utils.py:124] Killing local vLLM worker processes

/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/_distutils_hack/__init__.py:54: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
Traceback (most recent call last):
  File "/mnt/data/taoshuchang.tsc/IR_RAG/IRM/inference_new.py", line 299, in <module>
    main()
  File "/mnt/data/taoshuchang.tsc/IR_RAG/IRM/inference_new.py", line 291, in main
    evaluate_retrieval(
  File "/mnt/data/taoshuchang.tsc/IR_RAG/IRM/inference_new.py", line 236, in evaluate_retrieval
    llm = load_model(model_path, enable_lora=enable_lora)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/IR_RAG/IRM/inference_new.py", line 49, in load_model
    llm = LLM(model=model_path, tensor_parallel_size=torch.cuda.device_count(), enable_lora=enable_lora)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 214, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 564, in from_engine_args
    engine = cls(
             ^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 325, in __init__
    self.model_executor = executor_class(
                          ^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
    super().__init__(*args, **kwargs)
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 47, in __init__
    self._init_executor()
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_gpu_executor.py", line 110, in _init_executor
    self._run_workers("init_device")
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/executor/multiproc_gpu_executor.py", line 185, in _run_workers
    driver_worker_output = driver_worker_method(*args, **kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 176, in init_device
    init_worker_distributed_environment(self.parallel_config, self.rank,
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/worker/worker.py", line 448, in init_worker_distributed_environment
    init_distributed_environment(parallel_config.world_size, rank,
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/vllm/distributed/parallel_state.py", line 946, in init_distributed_environment
    torch.distributed.init_process_group(
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
    func_return = func(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1520, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/distributed/rendezvous.py", line 221, in _tcp_rendezvous_handler
    store = _create_c10d_store(
            ^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/taoshuchang.tsc/anaconda3/envs/py311llama33/lib/python3.11/site-packages/torch/distributed/rendezvous.py", line 189, in _create_c10d_store
    return TCPStore(
           ^^^^^^^^^
torch.distributed.DistStoreError: Timed out after 601 seconds waiting for clients. 1/8 clients joined.

ERROR conda.cli.main_run:execute(49): `conda run python -u /mnt/data/taoshuchang.tsc/IR_RAG/IRM/inference_new.py --test_data_path /mnt/data/taoshuchang.tsc/IR_RAG/IRM/datasets/hotpot_contriever/hotpot-dev.json --batch_size 4 --model_path /mnt/data/taoshuchang.tsc/IR_RAG/ckpt/hotpot_contriever/analyze_merge//hotpot_1doc_other_Meta-Llama-3-70B-Instruct_lr1e5 --result_save_path /mnt/data/taoshuchang.tsc/IR_RAG/IRM/result/hotpot_contriever/analyze/hotpot_1doc_other_Meta-Llama-3-70B-Instruct_lr1e5.json --batch True` failed. (See above for error)

```

Thank you for your help in resolving this issue.

cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o