Repository: huggingface/lighteval
Issue: [BUG] Tensor Paralell Doesn't Seem to Work
Root cause: Configuration Error

## Describe the bug
Trying qwq-32b on 8x h100s alongside Open-R1 repo's evals.

Running in dp=8 does not have enough memory, OOMs. Trying data parallel fails with cuda issues.

Causes this error that fails the eval:
```
(VllmWorkerProcess pid=1045470) [2025-03-08 05:38:28,352] [   ERROR]: Exception in worker VllmWorkerProcess while processing method init_device. (multiproc_worker_utils.py:240)
(VllmWorkerProcess pid=1045470) Traceback (most recent call last):
(VllmWorkerProcess pid=1045470)   File "/home/teknium/open-r1/venv/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 234, in _run_worker_process
(VllmWorkerProcess pid=1045470)     output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=1045470)   File "/home/teknium/open-r1/venv/lib/python3.10/site-packages/vllm/utils.py", line 2208, in run_method
(VllmWorkerProcess pid=1045470)     return func(*args, **kwargs)
(VllmWorkerProcess pid=1045470)   File "/home/teknium/open-r1/venv/lib/python3.10/site-packages/vllm/worker/worker.py", line 154, in init_device
(VllmWorkerProcess pid=1045470)     torch.cuda.set_device(self.device)
(VllmWorkerProcess pid=1045470)   File "/home/teknium/open-r1/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 478, in set_device
(VllmWorkerProcess pid=1045470)     torch._C._cuda_setDevice(device)
(VllmWorkerProcess pid=1045470)   File "/home/teknium/open-r1/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
(VllmWorkerProcess pid=1045470)     raise RuntimeError(
(VllmWorkerProcess pid=1045470) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```  


## To Reproduce
Install open-r1 repo and rurn the provided script in the readme:

```bash
MODEL=Qwen/QwQ-32B
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,tensor_parallel_size=8,data_parallel_size=1,max_model_length=32768,gpu_memory_utilisation=0.98"
OUTPUT_DIR=data/evals/$MODEL

# MATH-500
TASK=math_500
lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR```  
  


## Expected behavior
Run the eval 

## Version info

Ubuntu, 8x h100 system

Package                           Version       Editable project location
--------------------------------- ------------- -------------------------
absl-py                           2.1.0
accelerate                        1.3.0
aenum                             3.1.15
aiohappyeyeballs                  2.4.6
aiohttp                           3.11.12
aiohttp-cors                      0.7.0
aiosignal                         1.3.2
airportsdata                      20241001
annotated-types                   0.7.0
antlr4-python3-runtime            4.13.2
anyio                             4.8.0
astor                             0.8.1
async-timeout                     5.0.1
attrs                             25.1.0
bitsandbytes                      0.45.2
blake3                            1.0.4
blis                              0.7.11
cachetools                        5.5.1
catalogue                         2.0.10
certifi                           2025.1.31
chardet                           5.2.0
charset-normalizer                3.4.1
click                             8.1.8
cloudpathlib                      0.16.0
cloudpickle                       3.1.1
colorama                          0.4.6
colorful                          0.5.6
colorlog                          6.9.0
compressed-tensors                0.9.0
confection                        0.1.5
cymem                             2.0.11
DataProperty                      1.1.0
datasets                          3.2.0
deepspeed                         0.15.4
depyf                             0.18.0
dill                              0.3.8
diskcache                         5.6.3
distlib                           0.3.9
distro                            1.9.0
docker-pycreds                    0.4.0
einops                            0.8.1
exceptiongroup                    1.2.2
fastapi                           0.115.8
filelock                          3.17.0
flake8                            7.1.1
flash_attn                        2.7.4.post1
frozenlist                        1.5.0
fsspec                            2024.9.0
gguf                              0.10.0
gitdb                             4.0.12
GitPython                         3.1.44
google-api-core                   2.24.1
google-auth                       2.38.0
googleapis-common-protos          1.66.0
grpcio                            1.70.0
h11                               0.14.0
hf_transfer                       0.1.9
hjson                             3.1.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.28.1
idna                              3.10
importlib_metadata                8.6.1
iniconfig                         2.0.0
inquirerpy                        0.3.4
interegular                       0.3.3
isort                             6.0.0
Jinja2                            3.1.5
jiter                             0.8.2
joblib                            1.4.2
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
langcodes                         3.5.0
language_data                     1.3.0
lark                              1.2.2
latex2sympy2_extended             1.0.6
liger_kernel                      0.5.2
lighteval                         0.6.0.dev0
lm-format-enforcer                0.10.9
lxml                              5.3.1
marisa-trie                       1.2.1
markdown-it-py                    3.0.0
MarkupSafe                        3.0.2
math-verify                       0.5.2
mbstrdecoder                      1.1.4
mccabe                            0.7.0
mdurl                             0.1.2
mistral_common                    1.5.3
more-itertools                    10.6.0
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.19.0
multidict                         6.1.0
multiprocess                      0.70.16
murmurhash                        1.0.12
nest-asyncio                      1.6.0
networkx                          3.4.2
ninja                             1.11.1.3
nltk                              3.9.1
numpy                             1.26.4
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-ml-py                      12.570.86
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
open-r1                           0.1.0.dev0    /home/teknium/open-r1/src
openai                            1.61.1
opencensus                        0.11.4
opencensus-context                0.1.3
opencv-python-headless            4.11.0.86
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         24.2
pandas                            2.2.3
parameterized                     0.9.0
partial-json-parser               0.2.1.1.post5
pathvalidate                      3.2.3
pfzy                              0.3.4
pillow                            11.1.0
pip                               22.0.2
platformdirs                      4.3.6
pluggy                            1.5.0
portalocker                       3.1.1
preshed                           3.0.9
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.0.2
prompt_toolkit                    3.0.50
propcache                         0.2.1
proto-plus                        1.26.0
protobuf                          3.20.3
psutil                            6.1.1
py-cpuinfo                        9.0.0
py-spy                            0.4.0
pyarrow                           19.0.0
pyasn1                            0.6.1
pyasn1_modules                    0.4.1
pybind11                          2.13.6
pycodestyle                       2.12.1
pycountry                         24.6.1
pydantic                          2.10.6
pydantic_core                     2.27.2
pyflakes                          3.2.0
Pygments                          2.19.1
pytablewriter                     1.2.1
pytest                            8.3.4
python-dateutil                   2.9.0.post0
python-dotenv                     1.0.1
pytz                              2025.1
PyYAML                            6.0.2
pyzmq                             26.2.1
ray                               2.42.0
referencing                       0.36.2
regex                             2024.11.6
requests                          2.32.3
rich                              13.9.4
rouge-score                       0.1.2
rpds-py                           0.22.3
rsa                               4.9
ruff                              0.9.6
sacrebleu                         2.5.1
safetensors                       0.5.2
scikit-learn                      1.6.1
scipy                             1.15.1
sentencepiece                     0.2.0
sentry-sdk                        2.20.0
setproctitle                      1.3.4
setuptools                        59.6.0
six                               1.17.0
smart-open                        6.4.0
smmap                             5.0.2
sniffio                           1.3.1
spacy                             3.7.2
spacy-legacy                      3.0.12
spacy-loggers                     1.0.5
srsly                             2.5.1
starlette                         0.45.3
sympy                             1.13.1
tabledata                         1.3.4
tabulate                          0.9.0
tcolorpy                          0.1.7
termcolor                         2.3.0
thinc                             8.2.5
threadpoolctl                     3.5.0
tiktoken                          0.8.0
tokenizers                        0.21.0
tomli                             2.2.1
torch                             2.5.1
torchaudio                        2.5.1
torchvision                       0.20.1
tqdm                              4.67.1
transformers                      4.49.0.dev0
triton                            3.1.0
trl                               0.15.0.dev0
typepy                            1.3.4
typer                             0.9.4
typing_extensions                 4.12.2
tzdata                            2025.1
urllib3                           2.3.0
uvicorn                           0.34.0
uvloop                            0.21.0
virtualenv                        20.29.2
vllm                              0.7.1
wandb                             0.19.6
wasabi                            1.1.3
watchfiles                        1.0.4
wcwidth                           0.2.13
weasel                            0.3.4
websockets                        14.2
wheel                             0.45.1
wrapt                             1.17.2
xformers                          0.0.28.post3
xgrammar                          0.1.11
xxhash                            3.5.0
yarl                              1.18.3
zipp                              3.21.0



Thanks to @hynky1999 on twitter, 
`export VLLM_WORKER_MULTIPROC_METHOD=spawn`

seems to fix it.

source: https://x.com/HKydlicek/status/1898373521325117447