Repository: huggingface/lighteval
Issue: [BUG] VLLM Code Generate hangs
Root cause: Resource Mishandling

## Describe the bug
When running vllm lcb generate, it looks like generation completes but it just hangs:


```(VllmWorkerProcess pid=1325214) INFO 08-22 01:44:02 [model_runner.py:1671] Graph capturing finished in 17 secs, took 0.38 GiB
(VllmWorkerProcess pid=1325213) INFO 08-22 01:44:02 [model_runner.py:1671] Graph capturing finished in 17 secs, took 0.38 GiB
(VllmWorkerProcess pid=1325212) INFO 08-22 01:44:02 [model_runner.py:1671] Graph capturing finished in 17 secs, took 0.38 GiB
[2025-08-22 01:44:02,664] [    INFO]: Graph capturing finished in 17 secs, took 0.39 GiB (model_runner.py:1671)
[2025-08-22 01:44:02,665] [    INFO]: init engine (profile, create kv cache, warmup model) took 58.78 seconds (llm_engine.py:428)
[2025-08-22 01:44:21,275] [    INFO]: --- INIT SEEDS --- (pipeline.py:258)
[2025-08-22 01:44:21,275] [    INFO]: --- LOADING TASKS --- (pipeline.py:212)
[2025-08-22 01:44:21,275] [    INFO]: Found 1 custom tasks in /proj/checkpoints/shared_data/envs/code_eval/lib/python3.10/site-packages/lighteval/tasks/extended/ifeval/main.py (registry.py:142)
[2025-08-22 01:44:21,275] [    INFO]: Found 6 custom tasks in /proj/checkpoints/shared_data/envs/code_eval/lib/python3.10/site-packages/lighteval/tasks/extended/tiny_benchmarks/main.py (registry.py:142)
[2025-08-22 01:44:21,275] [    INFO]: Found 1 custom tasks in /proj/checkpoints/shared_data/envs/code_eval/lib/python3.10/site-packages/lighteval/tasks/extended/mt_bench/main.py (registry.py:142)
[2025-08-22 01:44:21,275] [    INFO]: Found 4 custom tasks in /proj/checkpoints/shared_data/envs/code_eval/lib/python3.10/site-packages/lighteval/tasks/extended/mix_eval/main.py (registry.py:142)
[2025-08-22 01:44:21,275] [    INFO]: Found 5 custom tasks in /proj/checkpoints/shared_data/envs/code_eval/lib/python3.10/site-packages/lighteval/tasks/extended/olympiade_bench/main.py (registry.py:142)
[2025-08-22 01:44:21,275] [    INFO]: Found 1 custom tasks in /proj/checkpoints/shared_data/envs/code_eval/lib/python3.10/site-packages/lighteval/tasks/extended/hle/main.py (registry.py:142)
[2025-08-22 01:44:21,275] [    INFO]: Found 23 custom tasks in /proj/checkpoints/shared_data/envs/code_eval/lib/python3.10/site-packages/lighteval/tasks/extended/lcb/main.py (registry.py:142)
[2025-08-22 01:44:21,292] [    INFO]: livecodebench/code_generation_lite v4_v5 (lighteval_task.py:187)
[2025-08-22 01:44:21,292] [ WARNING]: Careful, the task extended|lcb:codegeneration is using evaluation data to build the few shot examples. (lighteval_task.py:260)
[2025-08-22 01:44:46,703] [    INFO]: --- RUNNING MODEL --- (pipeline.py:482)
[2025-08-22 01:44:46,703] [    INFO]: Running RequestType.GREEDY_UNTIL requests (pipeline.py:468)
[2025-08-22 01:44:46,904] [ WARNING]: You cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring. (data.py:237)
Adding requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 268/268 [00:00<00:00, 572.17it/s]
Adding requests:  83%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                | 222/268 [00:00<00:00, 577.38it/s]
Processed prompts:  92%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌          | 3936/4288 [21:52<03:17,  1.78it/s, est. speed input: 1674.62 toks/s, output: 1539.18 toks/s]


Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 4288/4288 [51:08<00:00,  1.40it/s, est. speed input: 796.88 toks/s, output: 1017.20 toks/s]Splits: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████Splits: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [51:09<00:00, 3069.67s/it]

```

## To Reproduce

Pull from main as of today: 0e06249d1caa3c1d27a93e590929531291c9c493

#!/bin/bash

export CUDA_VISIBLE_DEVICES=0,1,2,3
export NCCL_IGNORE_DISABLED_P2P=1
export VLLM_SKIP_P2P_CHECK=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn

NUM_GPUS=4
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,tensor_parallel_size=$NUM_GPUS,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
TASK=aime24
OUTPUT_DIR=data/evals/$MODEL

export VLLM_WORKER_MULTIPROC_METHOD=spawn
lighteval vllm $MODEL_ARGS  "extended|lcb:codegeneration|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR

## Expected behavior

Results printed out and saved to output dir.

## Version info

Commit: 0e06249d1caa3c1d27a93e590929531291c9c493

transformers==4.52.3
datasets==3.6.0
vllm==0.9.2
torch==2.7.0
torchaudio==2.7.0
torchvision==0.22.0