Repository: huggingface/lighteval
Issue: [BUG] vLLM backend hangs with DDP
Root cause: External Dependency Breakage

## Describe the bug
When running the latest version of `vllm` (0.8.3), the backend hangs when evaluating with more than 1 GPU.

## To Reproduce

First install `vllm`:

```
pip install vllm==0.8.3
```

Then run:

```
lighteval vllm "pretrained=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B,dtype=bfloat16,data_parallel_size=2,max_model_length=32768,max_num_batched_tokens=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}" "lighteval|$TASK|0|0"     --use-chat-template     --output-dir $OUTPUT_DIR
```

The logs will hang at the Ray placement group part:

```
[2025-04-13 19:07:39,715] [    INFO]: PyTorch version 2.6.0 available. (config.py:54)
INFO 04-13 19:07:48 [__init__.py:239] Automatically detected platform cuda.
[2025-04-13 19:07:51,066] [    INFO]: --- LOADING MODEL --- (pipeline.py:189)
[2025-04-13 19:07:51,529] [    INFO]: --- INIT SEEDS --- (pipeline.py:263)
[2025-04-13 19:07:51,529] [    INFO]: --- LOADING TASKS --- (pipeline.py:216)
[2025-04-13 19:07:51,529] [    INFO]: Found 1 custom tasks in /fsx/lewis/git/hf/lighteval/src/lighteval/tasks/extended/ifeval/main.py (registry.py:142)
[2025-04-13 19:07:51,529] [    INFO]: Found 6 custom tasks in /fsx/lewis/git/hf/lighteval/src/lighteval/tasks/extended/tiny_benchmarks/main.py (registry.py:142)
[2025-04-13 19:07:51,529] [    INFO]: Found 1 custom tasks in /fsx/lewis/git/hf/lighteval/src/lighteval/tasks/extended/mt_bench/main.py (registry.py:142)
[2025-04-13 19:07:51,529] [    INFO]: Found 4 custom tasks in /fsx/lewis/git/hf/lighteval/src/lighteval/tasks/extended/mix_eval/main.py (registry.py:142)
[2025-04-13 19:07:51,529] [    INFO]: Found 5 custom tasks in /fsx/lewis/git/hf/lighteval/src/lighteval/tasks/extended/olympiade_bench/main.py (registry.py:142)
[2025-04-13 19:07:51,530] [    INFO]: Found 1 custom tasks in /fsx/lewis/git/hf/lighteval/src/lighteval/tasks/extended/hle/main.py (registry.py:142)
[2025-04-13 19:07:51,530] [    INFO]: Found 21 custom tasks in /fsx/lewis/git/hf/lighteval/src/lighteval/tasks/extended/lcb/main.py (registry.py:142)
[2025-04-13 19:07:51,534] [    INFO]: HuggingFaceH4/aime_2024 default (lighteval_task.py:187)
[2025-04-13 19:07:51,534] [ WARNING]: Careful, the task lighteval|aime24 is using evaluation data to build the few shot examples. (lighteval_task.py:260)
[2025-04-13 19:07:52,888] [    INFO]: --- RUNNING MODEL --- (pipeline.py:468)
[2025-04-13 19:07:52,888] [    INFO]: Running RequestType.GREEDY_UNTIL requests (pipeline.py:472)
[2025-04-13 19:07:52,906] [ WARNING]: You cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring. (data.py:260)
Splits:   0%|                                                                                                                                                                         | 0/2 [00:00<?, ?it/s][2025-04-13 19:07:52,915] [ WARNING]: context_size + max_new_tokens=33238 which is greater than self.max_length=32768. Truncating context to 0 tokens. (vllm_model.py:274)
[2025-04-13 19:07:55,258] [    INFO]: Started a local Ray instance. (worker.py:1841)
(pid=1276189) INFO 04-13 19:08:03 [__init__.py:239] Automatically detected platform cuda.
(run_inference_one_model pid=1276191) INFO 04-13 19:08:16 [config.py:600] This model supports multiple tasks: {'embed', 'score', 'generate', 'classify', 'reward'}. Defaulting to 'generate'.
(pid=1276191) INFO 04-13 19:08:03 [__init__.py:239] Automatically detected platform cuda.
(run_inference_one_model pid=1276191) INFO 04-13 19:08:16 [config.py:1780] Chunked prefill is enabled with max_num_batched_tokens=32768.
(run_inference_one_model pid=1276189) INFO 04-13 19:08:16 [config.py:600] This model supports multiple tasks: {'reward', 'generate', 'embed', 'classify', 'score'}. Defaulting to 'generate'.
(run_inference_one_model pid=1276191) INFO 04-13 19:08:18 [core.py:61] Initializing a V1 LLM engine (v0.8.3) with config: model='deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B', speculative_config=None, tokenizer='deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, override_neuron_config=None, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
(run_inference_one_model pid=1276191) INFO 04-13 19:08:18 [ray_utils.py:288] Ray is already initialized. Skipping Ray initialization.
(run_inference_one_model pid=1276191) INFO 04-13 19:08:18 [ray_utils.py:335] No current placement group found. Creating a new placement group.
```

## Expected behavior
No hanging.

## Version info

Running `lighteval@cc95ff274718186f587500556d7001645a273ce8`. Additional info:

```
- `transformers` version: 4.51.2
- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.11.11
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.5.3
- Accelerate version: 1.6.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3
```

same question when using vllm=0.8.2

Hi folks! I'm on it - the ray/vllm combo is causing incompatibilities across versions of vllm, so we'll try to manage the DP natively on the lighteval side. 

It will take some time however.

FYI @clefourrier this is how vLLM recommends doing data parallel: https://docs.vllm.ai/en/latest/getting_started/examples/data_parallel.html#data-parallel

Thanks a lot! I've been using it (and your convo on slack) as ref so far - it has 2 limitations for our use case:
- it would require us to change the pipeline level to manage the multiprocess
- they don't care about gathering in this example (whereas for evals we'll need to gather results to compute metrics)

Still wrapping my head on how to do this neatly ^^ but we used to do DP by hand at the start of lighteval so I'll probably do it like this

I see. I wonder then if it's possible to spin up N copies of vLLM's OpenAI server and then load balance it with a proxy like LiteLLM: https://docs.litellm.ai/docs/proxy/load_balancing ?

Not sure if that adds a lot of complexity though to the `lighteval` internals

any workaround for this issue? it blocks me on generation throughput.

Seems like a non ugly fix for this would mean using the AsyncVLLMEngine, @lewtun would that work for you? 

Is there any way I can help resolve this/any avenues I could help investigate? Not being able to parallelize evals is slowing my experiments quite a bit, and DDP still works with LM-Eval so I have been falling back on that, but it would be nice to be able to stay within the HuggingFace stack 😅

> Seems like a non ugly fix for this would mean using the AsyncVLLMEngine, [@lewtun](https://github.com/lewtun) would that work for you?

Yes that would be fine since it's not user facing right?

We'll likely be missing the logprobs

In the meantime @Neelectric you can get the same behavior as the harness by pinning ray/vllm to the same versions as they do since our current code is very similar. The issue is that we allow more flexibility in ray/vllm versions and they often get incompatible with one another 

FYI I just tested the latest version of vLLM (`v0.8.4`) and data parallel works again!

E.g. this works using `lighteval@989f5f5586de1ddfeceb0dfa5076bd0740d376fa`:

```
lighteval vllm "model_name=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B,data_parallel_size=8,dtype=bfloat16,max_model_length=32768,max_num_batched_tokens=32768,gpu_memory_utilization=0.8,generation_param
eters={max_new_tokens:32768,temperature:0.6,top_p:0.95}" "lighteval|gpqa:diamond|0|0"     --use-chat-template     --output-dir $OUTPUT_DIR
```

So if we bump the min required version, no changes are needed on the `lighteval` core

However I am seeing some rather large differences between DP and TP, but will post those in a separate issue

I'll pin v0.8.4 + ray but still add the async model (I'm almost done anyway) as a backup option

Which ray are you using?

Here's my full env as a reference

```
absl-py==2.2.2
accelerate==1.4.0
aenum==3.1.15
aiohappyeyeballs==2.6.1
aiohttp==3.11.18
aiosignal==1.3.2
airportsdata==20250224
annotated-types==0.7.0
antlr4-python3-runtime==4.13.2
anyio==4.9.0
astor==0.8.1
attrs==25.3.0
bitsandbytes==0.45.5
blake3==1.0.4
cachetools==5.5.2
certifi==2025.4.26
chardet==5.2.0
charset-normalizer==3.4.1
click==8.1.8
cloudpickle==3.1.1
colorama==0.4.6
colorlog==6.9.0
compressed-tensors==0.9.3
cupy-cuda12x==13.4.1
dataproperty==1.1.0
datasets==3.5.0
deepspeed==0.16.7
deprecated==1.2.18
depyf==0.18.0
dill==0.3.8
diskcache==5.6.3
distro==1.9.0
dnspython==2.7.0
docker-pycreds==0.4.0
e2b==1.3.3
e2b-code-interpreter==1.2.0
einops==0.8.1
email-validator==2.2.0
fastapi==0.115.12
fastapi-cli==0.0.7
fastrlock==0.8.3
filelock==3.18.0
flake8==7.2.0
flash-attn==2.7.4.post1
frozenlist==1.6.0
fsspec==2024.12.0
gguf==0.16.2
gitdb==4.0.12
gitpython==3.1.44
googleapis-common-protos==1.70.0
grpcio==1.71.0
h11==0.16.0
hf-transfer==0.1.9
hf-xet==1.0.5
hjson==3.1.0
httpcore==1.0.9
httptools==0.6.4
httpx==0.27.2
huggingface-hub==0.30.2
idna==3.10
importlib-metadata==8.0.0
iniconfig==2.1.0
inquirerpy==0.3.4
interegular==0.3.3
isort==6.0.1
jinja2==3.1.6
jiter==0.9.0
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2025.4.1
langdetect==1.0.9
lark==1.2.2
latex2sympy2-extended==1.0.6
liger-kernel==0.5.8
lighteval @ git+https://github.com/huggingface/lighteval.git@989f5f5586de1ddfeceb0dfa5076bd0740d376fa
llguidance==0.7.19
llvmlite==0.44.0
lm-format-enforcer==0.10.11
lxml==5.4.0
markdown-it-py==3.0.0
markupsafe==3.0.2
math-verify==0.5.2
mbstrdecoder==1.1.4
mccabe==0.7.0
mdurl==0.1.2
mistral-common==1.5.4
mpmath==1.3.0
msgpack==1.1.0
msgspec==0.19.0
multidict==6.4.3
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.4.2
ninja==1.11.1.4
nltk==3.9.1
numba==0.61.2
numpy==1.26.4
nvidia-cublas-cu12==12.4.5.8
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.1.3
nvidia-curand-cu12==10.3.5.147
nvidia-cusolver-cu12==11.6.1.9
nvidia-cusparse-cu12==12.3.1.170
nvidia-cusparselt-cu12==0.6.2
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
-e file:///fsx/lewis/git/hf/dev/open-r1
openai==1.76.0
opencv-python-headless==4.11.0.86
opentelemetry-api==1.26.0
opentelemetry-exporter-otlp==1.26.0
opentelemetry-exporter-otlp-proto-common==1.26.0
opentelemetry-exporter-otlp-proto-grpc==1.26.0
opentelemetry-exporter-otlp-proto-http==1.26.0
opentelemetry-proto==1.26.0
opentelemetry-sdk==1.26.0
opentelemetry-semantic-conventions==0.47b0
opentelemetry-semantic-conventions-ai==0.4.3
outlines==0.1.11
outlines-core==0.1.26
packaging==25.0
pandas==2.2.3
parameterized==0.9.0
partial-json-parser==0.2.1.1.post5
pathvalidate==3.2.3
pfzy==0.3.4
pillow==11.2.1
pip==25.0.1
platformdirs==4.3.7
pluggy==1.5.0
portalocker==3.1.1
prometheus-client==0.21.1
prometheus-fastapi-instrumentator==7.1.0
prompt-toolkit==3.0.51
propcache==0.3.1
protobuf==3.20.3
psutil==7.0.0
py-cpuinfo==9.0.0
pyarrow==19.0.1
pycodestyle==2.13.0
pycountry==24.6.1
pydantic==2.11.3
pydantic-core==2.33.1
pyflakes==3.3.2
pygments==2.19.1
pytablewriter==1.2.1
pytest==8.3.5
python-dateutil==2.9.0.post0
python-dotenv==1.1.0
python-json-logger==3.3.0
python-multipart==0.0.20
pytz==2025.2
pyyaml==6.0.2
pyzmq==26.4.0
ray==2.43.0
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
rich==14.0.0
rich-toolkit==0.14.3
rouge-score==0.1.2
rpds-py==0.24.0
ruff==0.11.7
sacrebleu==2.5.1
safetensors==0.5.3
scikit-learn==1.6.1
scipy==1.15.2
sentencepiece==0.2.0
sentry-sdk==2.27.0
setproctitle==1.3.5
setuptools==79.0.1
shellingham==1.5.4
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
starlette==0.46.2
sympy==1.13.1
tabledata==1.3.4
tabulate==0.9.0
tcolorpy==0.1.7
termcolor==2.3.0
threadpoolctl==3.6.0
tiktoken==0.9.0
tokenizers==0.21.1
torch==2.6.0
torchaudio==2.6.0
torchvision==0.21.0
tqdm==4.67.1
transformers @ git+https://github.com/huggingface/transformers.git@acdbe627e323dbc822f21499fead789b439cf45b
triton==3.2.0
trl @ git+https://github.com/huggingface/trl.git@1bca49515ecd5b85d16e68c42c76670e252e19f1
typepy==1.3.4
typer==0.15.2
typing-extensions==4.13.2
typing-inspection==0.4.0
tzdata==2025.2
urllib3==2.4.0
uvicorn==0.34.2
uvloop==0.21.0
vllm==0.8.4
wandb==0.19.10
watchfiles==1.0.5
wcwidth==0.2.13
websockets==15.0.1
wrapt==1.17.2
xformers==0.0.29.post2
xgrammar==0.1.18
xxhash==3.5.0
yarl==1.20.0
zipp==3.21.0
```


I'm having the same problem. I tried updating to the latest versions of ray (2.43.0) and vllm (0.8.4) as @lewton has but I'm still getting the same problem. Here's my full env:
```
$ pip freeze
absl-py==2.2.2
accelerate==1.4.0
aenum==3.1.15
aiohappyeyeballs==2.6.1
aiohttp==3.11.18
aiosignal==1.3.2
airportsdata==20250224
annotated-types==0.7.0
antlr4-python3-runtime==4.13.2
anyio==4.9.0
astor==0.8.1
attrs==25.3.0
bitsandbytes==0.45.5
blake3==1.0.4
cachetools==5.5.2
certifi==2025.4.26
chardet==5.2.0
charset-normalizer==3.4.1
click==8.1.8
cloudpickle==3.1.1
colorama==0.4.6
colorlog==6.9.0
compressed-tensors==0.9.3
cupy-cuda12x==13.4.1
DataProperty==1.1.0
datasets==3.5.1
deepspeed==0.16.7
Deprecated==1.2.18
depyf==0.18.0
dill==0.3.8
diskcache==5.6.3
distro==1.9.0
dnspython==2.7.0
docker-pycreds==0.4.0
e2b==1.3.3
e2b-code-interpreter==1.2.0
einops==0.8.1
email_validator==2.2.0
fastapi==0.115.12
fastapi-cli==0.0.7
fastrlock==0.8.3
filelock==3.18.0
flake8==7.2.0
flash_attn==2.7.4.post1
frozenlist==1.6.0
fsspec==2025.3.0
gguf==0.16.2
gitdb==4.0.12
GitPython==3.1.44
googleapis-common-protos==1.70.0
grpcio==1.71.0
h11==0.16.0
hf-xet==1.0.5
hf_transfer==0.1.9
hjson==3.1.0
httpcore==1.0.9
httptools==0.6.4
httpx==0.27.2
huggingface-hub==0.30.2
idna==3.10
importlib_metadata==8.0.0
iniconfig==2.1.0
inquirerpy==0.3.4
interegular==0.3.3
isort==6.0.1
Jinja2==3.1.6
jiter==0.9.0
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2025.4.1
langdetect==1.0.9
lark==1.2.2
latex2sympy2_extended==1.0.6
liger_kernel==0.5.8
lighteval @ git+https://github.com/huggingface/lighteval.git@bb14995c4eccab5cabd450b1e509c3c898a16921
llguidance==0.7.19
llvmlite==0.44.0
lm-format-enforcer==0.10.11
lxml==5.4.0
markdown-it-py==3.0.0
MarkupSafe==3.0.2
math-verify==0.5.2
mbstrdecoder==1.1.4
mccabe==0.7.0
mdurl==0.1.2
mistral_common==1.5.4
mpmath==1.3.0
msgpack==1.1.0
msgspec==0.19.0
multidict==6.4.3
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.4.2
ninja==1.11.1.4
nltk==3.9.1
numba==0.61.2
numpy==1.26.4
nvidia-cublas-cu12==12.4.5.8
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.1.3
nvidia-curand-cu12==10.3.5.147
nvidia-cusolver-cu12==11.6.1.9
nvidia-cusparse-cu12==12.3.1.170
nvidia-cusparselt-cu12==0.6.2
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
-e git+https://github.com/huggingface/open-r1.git@50590a41b9c3c97dcca9018a1778f8dd1645f525#egg=open_r1
openai==1.76.0
opencv-python-headless==4.11.0.86
opentelemetry-api==1.26.0
opentelemetry-exporter-otlp==1.26.0
opentelemetry-exporter-otlp-proto-common==1.26.0
opentelemetry-exporter-otlp-proto-grpc==1.26.0
opentelemetry-exporter-otlp-proto-http==1.26.0
opentelemetry-proto==1.26.0
opentelemetry-sdk==1.26.0
opentelemetry-semantic-conventions==0.47b0
opentelemetry-semantic-conventions-ai==0.4.3
outlines==0.1.11
outlines_core==0.1.26
packaging==25.0
pandas==2.2.3
parameterized==0.9.0
partial-json-parser==0.2.1.1.post5
pathvalidate==3.2.3
pfzy==0.3.4
pillow==11.2.1
platformdirs==4.3.7
pluggy==1.5.0
portalocker==3.1.1
prometheus-fastapi-instrumentator==7.1.0
prometheus_client==0.21.1
prompt_toolkit==3.0.51
propcache==0.3.1
protobuf==3.20.3
psutil==7.0.0
py-cpuinfo==9.0.0
pyarrow==20.0.0
pycodestyle==2.13.0
pycountry==24.6.1
pydantic==2.11.3
pydantic_core==2.33.1
pyflakes==3.3.2
Pygments==2.19.1
pytablewriter==1.2.1
pytest==8.3.5
python-dateutil==2.9.0.post0
python-dotenv==1.1.0
python-json-logger==3.3.0
python-multipart==0.0.20
pytz==2025.2
PyYAML==6.0.2
pyzmq==26.4.0
ray==2.43.0
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
rich==14.0.0
rich-toolkit==0.14.3
rouge_score==0.1.2
rpds-py==0.24.0
ruff==0.11.7
sacrebleu==2.5.1
safetensors==0.5.3
scikit-learn==1.6.1
scipy==1.15.2
sentencepiece==0.2.0
sentry-sdk==2.27.0
setproctitle==1.3.5
shellingham==1.5.4
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
starlette==0.46.2
sympy==1.13.1
tabledata==1.3.4
tabulate==0.9.0
tcolorpy==0.1.7
termcolor==2.3.0
threadpoolctl==3.6.0
tiktoken==0.9.0
tokenizers==0.21.1
torch==2.6.0
torchaudio==2.6.0
torchvision==0.21.0
tqdm==4.67.1
transformers @ git+https://github.com/huggingface/transformers.git@acdbe627e323dbc822f21499fead789b439cf45b
triton==3.2.0
trl @ git+https://github.com/huggingface/trl.git@1bca49515ecd5b85d16e68c42c76670e252e19f1
typepy==1.3.4
typer==0.15.3
typing-inspection==0.4.0
typing_extensions==4.13.2
tzdata==2025.2
urllib3==2.4.0
uvicorn==0.34.2
uvloop==0.21.0
vllm==0.8.4
wandb==0.19.10
watchfiles==1.0.5
wcwidth==0.2.13
websockets==15.0.1
wrapt==1.17.2
xformers==0.0.29.post2
xgrammar==0.1.18
xxhash==3.5.0
yarl==1.20.0
zipp==3.21.0
```

@akriegman try `export VLLM_ENABLE_V1_MULTIPROCESSING=0`
or just disable v1 engine

I also tried updating to the versions of ray (2.43.0) and vllm (0.8.4) as well as using the same lighteval version (lighteval @ git+https://github.com/huggingface/lighteval.git@989f5f5586de1ddfeceb0dfa5076bd0740d376fa) and it still hangs. The "export VLLM_ENABLE_V1_MULTIPROCESSING=0" does not help. 

I figured it out. In order to get it to work on one gpu I had to add `VLLM_WORKER_MULTIPROC_METHOD=spawn`, and then for some reason that also fixed the multi gpu case. Here's the working commands for both cases, modified from the Open-R1 readme:

```bash
function eval {
  NUM_GPUS=4
  MODEL=./data/Qwen2.5-1.5B-Open-R1-Distill
  MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,data_parallel_size=$NUM_GPUS,max_model_length=32768,max_num_batched_tokens=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
  TASK=aime24
  OUTPUT_DIR=data/evals/$MODEL

  VLLM_WORKER_MULTIPROC_METHOD=spawn lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
      --use-chat-template \
      --output-dir $OUTPUT_DIR
}

function eval_single {
  MODEL=./data/Qwen2.5-1.5B-Open-R1-Distill
  MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,max_num_batched_tokens=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
  OUTPUT_DIR=data/evals/$MODEL
  TASK=aime24

  VLLM_WORKER_MULTIPROC_METHOD=spawn lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
      --use-chat-template \
      --output-dir $OUTPUT_DIR
}
```

With the same dependencies that you listed earlier? Because still hanging for me :/ @akriegman 

Nearly. `setproctitle` and `openai` were both bumped one patch version but besides that the same :/

Great detective work @akriegman ! Adding that to my PR here to fix `open-r1`: https://github.com/huggingface/open-r1/pull/629/files Thanks :)

That's very neat @akriegman , do you want to add that to the lighteval documentation?

As a side note, the AsyncVLLM PR is ready if some of you folks want to test it locally and tell me wyt (ty for your review @lewtun )

closing because fixed with latest VLLM release.

@NathanHB Hi there, I have upgrade vllm to latest version (0.8.5.post1) as well as lighteval (0.9.2), but the problem is still there: 

```
No current placement group found. Creating a new placement group.
```

My ray version is 2.46.0. Could you please give a hint on which VLLM release should we use? It seems the latest vllm didn't solve the problem. 

mhhh, even after setting the env varibales mentioned above ?

> mhhh, even after setting the env varibales mentioned above ?

Yes, with the following script I can reproduce the problem: 

```
NUM_GPUS=3
MODEL=./data/Qwen2.5-1.5B-Open-R1-Distill
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,data_parallel_size=$NUM_GPUS,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
TASK=aime24
OUTPUT_DIR=./data/evals/$MODEL

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_HOST_IP=0.0.0.0

lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
      --use-chat-template \
      --output-dir $OUTPUT_DIR
```

And the deps are: 

```
$ uv pip list | grep ray
ray                                      2.46.0
$ uv pip list | grep vllm
vllm                                     0.8.5.post1
$ uv pip list | grep lighteval
lighteval                                0.9.1.dev0
```

These dependencies work for me:

```
- Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.31
- Python version: 3.11.11
- TRL version: 0.18.0.dev0
- PyTorch version: 2.6.0
- CUDA device(s): not available
- Transformers version: 4.52.0.dev0
- Accelerate version: 1.4.0
- Accelerate config: not found
- Datasets version: 3.5.1
- HF Hub version: 0.30.2
- bitsandbytes version: 0.45.5
- DeepSpeed version: 0.16.7
- Diffusers version: not installed
- Liger-Kernel version: 0.5.8
- LLM-Blender version: not installed
- OpenAI version: 1.76.2
- PEFT version: not installed
- vLLM version: 0.8.4
- LightEval commit: d3da6b9bbf38104c8b5e1acc86f83541f9a502d1
```

just fyi, i have a similar issue:
lighteval==0.8.1 ray==2.46.0 vllm==0.7.3 
when dp > 1, ray hangs with log like:
Error: No available node types can fulfill resource request {'GPU': 1.0, 'node:0.0.0.0': 0.001}

the solution is to set VLLM_HOST_IP env to, say, eth0 's ip address before vllm start. it does not work with '127.0.0.1'
i have not dig the root cause.

however, it breaks again with vllm 0.8.5, not dig it yet. related: https://github.com/vllm-project/vllm/issues/17863

Yes, the latest version of the code works for me. 

Python version: 3.11.12

```
accelerate                               1.4.0
datasets                                 3.6.0
flash-attn                               2.7.4.post1
liger-kernel                             0.5.10
lighteval                                0.9.1.dev0
ray                                      2.46.0
torch                                    2.6.0
transformers                             4.52.0.dev0
triton                                   3.2.0
trl                                      0.18.0.dev0
vllm                                     0.8.4
```

@Nativu5

```
NUM_GPUS=8

MODEL=data/OpenR1-Qwen-7B-SFT
TASK=aime24

MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,data_parallel_size=$NUM_GPUS,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

VLLM_WORKER_MULTIPROC_METHOD=spawn lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```