Repository: EleutherAI/lm-evaluation-harness
Issue: [BUG] A weird bug that VLLM hangs while initializing: "Worker ready; awaiting tasks" forever until timeout
Root cause: Resource Mishandling

I ran an MMLU evaluation using this sh:

```
export CUDA_VISIBLE_DEVICES="0,1,2,3"
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_LOGGING_LEVEL=DEBUG
export NCCL_DEBUG=TRACE
export VLLM_TRACE_FUNCTION=1

MODEL_PATH=/data1/model/qwen2_5-72b-instruct
TASK=lm_eval/tasks/mmlu/flan_cot_fewshot

accelerate launch -m lm_eval \
--model vllm \
--model_args pretrained=$MODEL_PATH,max_gen_toks=768,tensor_parallel_size=4,data_parallel_size=1 \
--tasks $TASK \
--num_fewshot 4 \
--output_path $OUT_PATH \
--log_samples \
--write_out \
--batch_size 1 \
--limit 40
```

The output log be like
```
[1;36m(VllmWorkerProcess pid=1876891)[0;0m INFO 06-29 19:49:29 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876887)[0;0m INFO 06-29 19:49:29 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876892)[0;0m INFO 06-29 19:49:29 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876893)[0;0m INFO 06-29 19:49:29 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876886)[0;0m INFO 06-29 19:49:30 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876900)[0;0m INFO 06-29 19:49:30 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876901)[0;0m INFO 06-29 19:49:30 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876905)[0;0m INFO 06-29 19:49:30 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876885)[0;0m DEBUG 06-29 19:49:30 parallel_state.py:845] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:33265 backend=nccl
[1;36m(VllmWorkerProcess pid=1876906)[0;0m INFO 06-29 19:49:30 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876899)[0;0m INFO 06-29 19:49:30 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876907)[0;0m INFO 06-29 19:49:30 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
[1;36m(VllmWorkerProcess pid=1876891)[0;0m DEBUG 06-29 19:49:31 parallel_state.py:845] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:36875 backend=nccl
[1;36m(VllmWorkerProcess pid=1876887)[0;0m DEBUG 06-29 19:49:31 parallel_state.py:845] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:33265 backend=nccl
[1;36m(VllmWorkerProcess pid=1876892)[0;0m DEBUG 06-29 19:49:31 parallel_state.py:845] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:36875 backend=nccl
[1;36m(VllmWorkerProcess pid=1876893)[0;0m DEBUG 06-29 19:49:31 parallel_state.py:845] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:36875 backend=nccl
[1;36m(VllmWorkerProcess pid=1876886)[0;0m DEBUG 06-29 19:49:31 parallel_state.py:845] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:33265 backend=nccl
[1;36m(VllmWorkerProcess pid=1876900)[0;0m DEBUG 06-29 19:49:32 parallel_state.py:845] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:45139 backend=nccl
[1;36m(VllmWorkerProcess pid=1876901)[0;0m DEBUG 06-29 19:49:33 parallel_state.py:845] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:45139 backend=nccl
[1;36m(VllmWorkerProcess pid=1876905)[0;0m DEBUG 06-29 19:49:33 parallel_state.py:845] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:37369 backend=nccl
[1;36m(VllmWorkerProcess pid=1876899)[0;0m DEBUG 06-29 19:49:33 parallel_state.py:845] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:45139 backend=nccl
[1;36m(VllmWorkerProcess pid=1876906)[0;0m DEBUG 06-29 19:49:33 parallel_state.py:845] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:37369 backend=nccl
[1;36m(VllmWorkerProcess pid=1876907)[0;0m DEBUG 06-29 19:49:33 parallel_state.py:845] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:37369 backend=nccl
```

I had located that it hangs in `row108 lm_eval/models/vllm_causallms.py: self.model = LLM(**self.model_args)`, which was initialized in `row205-212: lm_eval/evaluator.py: `:
```
lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
                model_args,
                {
                    "batch_size": batch_size,
                    "max_batch_size": max_batch_size,
                    "device": device,
                },
            )
```

So i printed the `model_args` out and tried to initialize a vllm model directly in a new python script, and it successed. Thus i'm sure that it is not the problem of vllm.

I also tried to remove ray (since i don't need dp), but the initialization of vllm still hangs.

That bug is extremely weird, the only thing i'm sure is that something before the vllm initialization makes it hangs, but i cannot find what it is. Hope i can get some help here

My pip list:
```
Package                           Version       Editable project location
--------------------------------- ------------- -------------------------------------------
absl-py                           2.1.0
accelerate                        0.31.0
adapters                          1.1.0
aiofiles                          23.2.1
aiohappyeyeballs                  2.4.4
aiohttp                           3.11.11
aiosignal                         1.3.2
alpaca_eval                       0.6.2
annotated-types                   0.7.0
antlr4-python3-runtime            4.7.2
anyio                             4.7.0
asttokens                         3.0.0
async-timeout                     5.0.1
attrs                             24.3.0
autoflake                         2.3.1
beaker-py                         1.32.3
bitsandbytes                      0.45.0
black                             24.10.0
blinker                           1.9.0
blis                              0.7.11
cached-property                   2.0.1
catalogue                         2.0.10
certifi                           2024.12.14
chardet                           5.2.0
charset-normalizer                3.4.0
click                             8.1.8
cloudpathlib                      0.20.0
cloudpickle                       3.1.0
colorama                          0.4.6
coloredlogs                       15.0.1
comm                              0.2.2
conda-pack                        0.8.1
confection                        0.1.5
contourpy                         1.3.1
cycler                            0.12.1
cymem                             2.0.10
DataProperty                      1.1.0
datasets                          3.2.0
debugpy                           1.8.14
decorator                         5.2.1
deepspeed                         0.15.0
dill                              0.3.8
diskcache                         5.6.3
distro                            1.9.0
docker                            7.1.0
docker-pycreds                    0.4.0
docopt                            0.6.2
einops                            0.8.0
elastic-transport                 8.15.1
elasticsearch                     8.6.2
et_xmlfile                        2.0.0
evaluate                          0.4.3
exceptiongroup                    1.2.2
executing                         2.2.0
fastapi                           0.115.6
ffmpy                             0.5.0
filelock                          3.16.1
fire                              0.7.0
flake8                            7.1.1
flash-attn                        2.6.3
Flask                             3.1.0
flatbuffers                       25.1.24
fonttools                         4.55.3
frozenlist                        1.5.0
fsspec                            2024.9.0
gguf                              0.9.1
gitdb                             4.0.11
GitPython                         3.1.43
gradio                            5.9.1
gradio_client                     1.5.2
grpcio                            1.68.1
h11                               0.14.0
hf_transfer                       0.1.8
hjson                             3.1.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.27.2
huggingface-hub                   0.27.0
humanfriendly                     10.0
idna                              3.10
immutabledict                     4.2.1
importlib_metadata                8.5.0
iniconfig                         2.0.0
interegular                       0.3.3
ipykernel                         6.29.5
ipython                           8.37.0
isort                             5.13.2
itsdangerous                      2.2.0
jedi                              0.19.2
Jinja2                            3.1.5
jiter                             0.8.2
joblib                            1.4.2
jsonlines                         4.0.0
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
jupyter_client                    8.6.3
jupyter_core                      5.8.1
kiwisolver                        1.4.7
langcodes                         3.5.0
langdetect                        1.0.9
language_data                     1.3.0
lark                              1.2.2
latex2sympy2                      1.9.1
litellm                           1.55.9
llvmlite                          0.43.0
lm_eval                           0.4.7        
lm-format-enforcer                0.10.6
lxml                              6.0.0
marisa-trie                       1.2.1
Markdown                          3.7
markdown-it-py                    3.0.0
MarkupSafe                        2.1.5
matplotlib                        3.10.0
matplotlib-inline                 0.1.7
mbstrdecoder                      1.1.4
mccabe                            0.7.0
mdurl                             0.1.2
mistral_common                    1.5.1
more-itertools                    10.7.0
mosestokenizer                    1.2.1
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.18.6
multidict                         6.1.0
multiprocess                      0.70.16
murmurhash                        1.0.11
mypy-extensions                   1.0.0
nest_asyncio                      1.6.0
networkx                          3.4.2
ninja                             1.11.1.3
nltk                              3.8.1
numba                             0.60.0
numexpr                           2.11.0
numpy                             1.26.4
nvidia-cublas-cu12                12.1.3.1
nvidia-cuda-cupti-cu12            12.1.105
nvidia-cuda-nvrtc-cu12            12.1.105
nvidia-cuda-runtime-cu12          12.1.105
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.0.2.54
nvidia-curand-cu12                10.3.2.106
nvidia-cusolver-cu12              11.4.5.107
nvidia-cusparse-cu12              12.1.0.106
nvidia-ml-py                      12.560.30
nvidia-nccl-cu12                  2.20.5
nvidia-nvjitlink-cu12             12.6.85
nvidia-nvtx-cu12                  12.1.105
onnxruntime                       1.20.1
open_instruct                     0.0.1         
openai                            1.58.1
openfile                          0.0.7
openpyxl                          3.1.5
orjson                            3.10.12
outlines                          0.0.46
packaging                         24.2
pandas                            2.2.3
parso                             0.8.4
partial-json-parser               0.2.1.1.post4
pathspec                          0.12.1
pathvalidate                      3.3.1
patsy                             1.0.1
Pebble                            5.1.1
peft                              0.14.0
pexpect                           4.9.0
pickleshare                       0.7.5
pillow                            10.4.0
pip                               24.3.1
platformdirs                      4.3.6
pluggy                            1.5.0
portalocker                       3.2.0
preshed                           3.0.9
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.0.0
prompt_toolkit                    3.0.51
propcache                         0.2.1
protobuf                          5.29.2
psutil                            6.1.1
ptyprocess                        0.7.0
pure_eval                         0.2.3
py-cpuinfo                        9.0.0
pyairports                        2.1.1
pyarrow                           18.1.0
pybind11                          2.13.6
pycodestyle                       2.12.1
pycountry                         24.6.1
pydantic                          2.10.4
pydantic_core                     2.27.2
pydub                             0.25.1
pyflakes                          3.2.0
Pygments                          2.18.0
pyparsing                         3.2.0
pytablewriter                     1.2.1
pytest                            8.3.4
python-dateutil                   2.9.0.post0
python-dotenv                     1.0.1
python-multipart                  0.0.20
pytz                              2024.2
PyYAML                            6.0.2
pyzmq                             26.2.0
ray                               2.40.0
referencing                       0.35.1
regex                             2024.11.6
requests                          2.32.3
rich                              13.9.4
rouge_score                       0.1.2
rpds-py                           0.22.3
ruff                              0.8.4
sacrebleu                         2.5.1
safehttpx                         0.1.6
safetensors                       0.4.5
scikit-learn                      1.6.0
scipy                             1.14.1
semantic-version                  2.10.0
sentencepiece                     0.2.0
sentry-sdk                        2.19.2
setproctitle                      1.3.4
setuptools                        69.5.1
shellingham                       1.5.4
six                               1.17.0
skops                             0.11.0
smart-open                        7.1.0
smmap                             5.0.1
sniffio                           1.3.1
spacy                             3.7.6
spacy-legacy                      3.0.12
spacy-loggers                     1.0.5
sqlitedict                        2.1.0
srsly                             2.5.0
stack_data                        0.6.3
starlette                         0.41.3
sympy                             1.12
tabledata                         1.3.4
tabulate                          0.9.0
tcolorpy                          0.1.7
tensorboard                       2.18.0
tensorboard-data-server           0.7.2
termcolor                         2.5.0
thinc                             8.2.5
threadpoolctl                     3.5.0
tiktoken                          0.7.0
timeout-decorator                 0.5.0
tokenizers                        0.21.0
tomli                             2.2.1
tomlkit                           0.13.2
toolwrapper                       2.1.0
torch                             2.4.0+cu121
torchaudio                        2.4.0+cu121
torchvision                       0.19.0+cu121
tornado                           6.5.1
tqdm                              4.67.1
tqdm-multiprocess                 0.0.11
traitlets                         5.14.3
transformers                      4.47.1
triton                            3.0.0
typepy                            1.3.4
typer                             0.15.1
typing_extensions                 4.12.2
tzdata                            2024.2
uctools                           1.3.0
unidic-lite                       1.0.8
urllib3                           2.3.0
uvicorn                           0.34.0
uvloop                            0.21.0
vllm                              0.6.1.post2
vllm-flash-attn                   2.6.1
wandb                             0.19.1
wasabi                            1.1.3
watchfiles                        1.0.3
wcwidth                           0.2.13
weasel                            0.4.1
websockets                        14.1
Werkzeug                          3.1.3
wheel                             0.45.1
word2number                       1.1
wrapt                             1.17.0
wtpsplit                          2.1.4
xformers                          0.0.27.post2
xxhash                            3.5.0
yarl                              1.18.3
zipp                              3.21.0
zstandard                         0.23.0
```
and the local cuda version is 12.1

don't using accelerate to launch vllm, stupid me