Repository: EleutherAI/lm-evaluation-harness
Issue: Hang with VLLM backend when data_parallel_size > 1
Root cause: External Dependency Breakage

```bash
lm_eval --model vllm \
        --model_args pretrained=$MODEL,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=4 \
        --tasks humaneval \
        --batch_size 32 \
        --confirm_run_unsafe_code \
        --predict_only \
        --output_path $PATH
```

```bash
Evaluating model: /mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/trained_model/llama3.2-3b-gem-412-lr2e5/checkpoint-14676
INFO 04-14 10:49:32 [__init__.py:239] Automatically detected platform cuda.
2025-04-14:10:53:14,475 INFO     [lm_eval.__main__:379] Selected Tasks: ['humaneval']
2025-04-14:10:53:14,506 INFO     [lm_eval.evaluator:177] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-04-14:10:53:14,506 INFO     [lm_eval.evaluator:214] Initializing vllm model, with arguments: {'pretrained': '/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/trained_model/qwen2.5-3b-ndp-402-wtemplate/checkpoint-14676', 'dtype': 'auto', 'gpu_memory_utilization': 0.8, 'data_parallel_size': 4}
2025-04-14:10:53:14,506 WARNING  [lm_eval.models.vllm_causallms:109] You might experience occasional issues with model weight downloading when data_parallel is in use. To ensure stable performance, run with data_parallel_size=1 until the weights are downloaded and cached.
2025-04-14:10:53:14,506 INFO     [lm_eval.models.vllm_causallms:114] Manual batching is not compatible with data parallelism.
Generating test split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 164/164 [00:00<00:00, 5400.19 examples/s]
2025-04-14:10:53:16,719 WARNING  [lm_eval.api.task:328] [Task: humaneval] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2025-04-14:10:53:16,719 WARNING  [lm_eval.api.task:328] [Task: humaneval] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2025-04-14:10:53:16,728 INFO     [lm_eval.evaluator:269] Processing humaneval in output-only mode. Metrics will not be calculated!
2025-04-14:10:53:16,729 INFO     [lm_eval.api.task:424] Building contexts for humaneval on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 164/164 [00:00<00:00, 3269.22it/s]
2025-04-14:10:53:16,787 INFO     [lm_eval.evaluator:525] Running generate_until requests
Running generate_until requests:   0%|                                                                                                                                                  | 0/164 [00:00<?, ?it/s]2025-04-14 10:53:19,610 INFO worker.py:1841 -- Started a local Ray instance.
(pid=36108) INFO 04-14 10:53:36 [__init__.py:239] Automatically detected platform cuda.
(run_inference_one_model pid=36136) INFO 04-14 10:53:57 [config.py:600] This model supports multiple tasks: {'reward', 'embed', 'generate', 'score', 'classify'}. Defaulting to 'generate'.
(pid=36109) INFO 04-14 10:53:36 [__init__.py:239] Automatically detected platform cuda. [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(run_inference_one_model pid=36114) INFO 04-14 10:53:57 [config.py:600] This model supports multiple tasks: {'embed', 'score', 'reward', 'generate', 'classify'}. Defaulting to 'generate'.
(run_inference_one_model pid=36114) INFO 04-14 10:53:57 [config.py:1780] Chunked prefill is enabled with max_num_batched_tokens=8192.
(run_inference_one_model pid=36109) INFO 04-14 10:53:57 [config.py:600] This model supports multiple tasks: {'classify', 'score', 'embed', 'generate', 'reward'}. Defaulting to 'generate'.
(run_inference_one_model pid=36108) INFO 04-14 10:53:57 [config.py:600] This model supports multiple tasks: {'embed', 'generate', 'reward', 'classify', 'score'}. Defaulting to 'generate'.
(run_inference_one_model pid=36108) INFO 04-14 10:53:59 [core.py:61] Initializing a V1 LLM engine (v0.8.3) with config: model='/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/trained_model/qwen2.5-3b-ndp-402-wtemplate/checkpoint-14676', speculative_config=None, tokenizer='/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/trained_model/qwen2.5-3b-ndp-402-wtemplate/checkpoint-14676', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/trained_model/qwen2.5-3b-ndp-402-wtemplate/checkpoint-14676, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
(run_inference_one_model pid=36108) INFO 04-14 10:53:59 [ray_utils.py:288] Ray is already initialized. Skipping Ray initialization.
(run_inference_one_model pid=36108) INFO 04-14 10:53:59 [ray_utils.py:335] No current placement group found. Creating a new placement group.
```
But with sglang backend everything works fine.

After a simple test, I found that the problem occurs with the data_parallel_size parameter. When it is greater than 1, it triggers an eternal hang.

Same Issue, did you figure this out?

> Same Issue, did you figure this out?同样的问题，你解决了吗？

not yet, this problem is caused by vllm itself https://github.com/vllm-project/vllm/issues/16588

Unusually, I found that setting `export VLLM_USE_V1=0` works for me. I am also using `VLLM_WORKER_MULTIPROC_METHOD=spawn` on vllm==0.8.5post1

Hey @wheynelau , this is a great find. Would you mind posting your comment on another issue on vLLM (https://github.com/vllm-project/vllm/issues/16588 ) to attract the community to fix the DP problem under the v1 engine? (Since v0 is already in stability maintenance phase.)

@rangehow Personally, I think they are quite different issues. I think the VLLM V1 data parallelism is not working yet: https://github.com/vllm-project/vllm/issues/1622, so they will have to fix that too. 

I think the best option for us is to use the V0, or use the `openai` endpoint. 

@baberabb could you consider adding these instructions? I added these to my code, happy to make a PR. Adding the below doesn't need any exporting, but happy to hear thoughts from the maintainers. 

```
+import os
 import copy
 import logging
 from importlib.metadata import version
@@ -111,6 +112,10 @@ class VLLM(TemplateLM):
             )
             self.model_args["distributed_executor_backend"] = "ray"
             self.batch_size = "auto"
+            eval_logger.info(
+                "VLLM V1 has some issues with data parallelism, setting V0 for now."
+            )
+            os.environ["VLLM_USE_V1"] = "0"
             eval_logger.info("Manual batching is not compatible with data parallelism.")
 
             from transformers import AutoConfig
```


> [@rangehow](https://github.com/rangehow) Personally, I think they are quite different issues. I think the VLLM V1 data parallelism is not working yet: [vllm-project/vllm#1622](https://github.com/vllm-project/vllm/pull/1622), so they will have to fix that too.
> 
> I think the best option for us is to use the V0, or use the `openai` endpoint.
> 
> [@baberabb](https://github.com/baberabb) could you consider adding these instructions? I added these to my code, happy to make a PR. Adding the below doesn't need any exporting, but happy to hear thoughts from the maintainers.
> 
> ```
> +import os
>  import copy
>  import logging
>  from importlib.metadata import version
> @@ -111,6 +112,10 @@ class VLLM(TemplateLM):
>              )
>              self.model_args["distributed_executor_backend"] = "ray"
>              self.batch_size = "auto"
> +            eval_logger.info(
> +                "VLLM V1 has some issues with data parallelism, setting V0 for now."
> +            )
> +            os.environ["VLLM_USE_V1"] = "0"
>              eval_logger.info("Manual batching is not compatible with data parallelism.")
>  
>              from transformers import AutoConfig
> ```

However, unfortunately, the maintainer of vLLM @DarkLight1337 just replied to me saying:  
"From my understanding, data parallelism is only available in V1."

Oh yes, I saw that reply too, I tried to fix the parallelism to get `lm-eval` to use the engine, I just realised I sent the wrong issue. The right one is here:

https://github.com/vllm-project/vllm/issues/16222

> `benchmarks/benchmark_throughput.py` does not support data parallel right now. offline inference with data parallel is quite different, you cannot just call one `LLM.generate`, but multiple instances need to call `LLM.generate` together. 

 _Originally posted by @youkaichao in [#16222](https://github.com/vllm-project/vllm/issues/16222#issuecomment-2798921446)_

Maybe its not as easy as just declaring a single vllm instance. But I'm still puzzled because we are not using the DP of V1, we are letting lm-eval handle the DP. 

 sorry that I initially assumed the data_parallel_size in lmeval was being passed to the LLM engine initialization. I just took a look at the code again. A possible issue could be whether it's caused by GPU preemption.  
We can try not relying on Ray's `num_gpus`, and instead manually set `CUDA_VISIBLE_DEVICES` inside the Actor based on the passed GPU ID. （just a guess, untested）

https://github.com/EleutherAI/lm-evaluation-harness/blob/fc5019ead53c45119c522c62e8eea2daa837c56e/lm_eval/models/vllm_vlms.py#L111-L131


```python
@ray.remote
def run_inference_one_model(
            model_args: dict, 
            sampling_params: SamplingParams, 
            requests_chunk: List[dict],
            gpu_id: int # Pass GPU ID
        ):
            # Set CUDA_VISIBLE_DEVICES *before* importing torch or initializing LLM
            os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu_id) 
```