Repository: EleutherAI/lm-evaluation-harness
Issue: vllm with tensor_parallel_mode is not working at all because of multiprocessing problem
Root cause: Resource Mishandling

`CUDA_VISIBLE_DEVICES=0,1 lm_eval --model vllm \
    --model_args pretrained=/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.80 \
    --tasks ai2_arc\
    --batch_size auto\
    --device cuda\`

**I run above setting, but below error has been occured.**

`⬢ [notebook] ❯ sh agieval.sh 
2024-10-28:11:14:24,795 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-28:11:14:30,767 INFO     [__main__.py:376] Selected Tasks: ['ai2_arc']
2024-10-28:11:14:30,770 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-28:11:14:30,770 INFO     [evaluator.py:201] Initializing vllm model, with arguments: {'pretrained': '/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct', 'tensor_parallel_size': 2, 'dtype': 'auto', 'gpu_memory_utilization': 0.8}
INFO 10-28 11:14:34 config.py:905] Defaulting to use mp for distributed inference
WARNING 10-28 11:14:34 arg_utils.py:957] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
INFO 10-28 11:14:34 config.py:1021] Chunked prefill is enabled with max_num_batched_tokens=512.
INFO 10-28 11:14:34 llm_engine.py:237] Initializing an LLM engine (v0.6.3.post1) with config: model='/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct', speculative_config=None, tokenizer='/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 10-28 11:14:35 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 56 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 10-28 11:14:35 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=479247) INFO 10-28 11:14:35 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=479247) INFO 10-28 11:14:35 utils.py:1008] Found nccl from library libnccl.so.2
INFO 10-28 11:14:35 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=479247) INFO 10-28 11:14:35 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 10-28 11:14:35 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 10-28 11:14:36 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/jovyan/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=479247) INFO 10-28 11:14:36 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/jovyan/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 10-28 11:14:36 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f807fb58150>, local_subscribe_port=45655, remote_subscribe_port=None)
INFO 10-28 11:14:36 model_runner.py:1056] Starting to load model /home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct...
(VllmWorkerProcess pid=479247) INFO 10-28 11:14:36 model_runner.py:1056] Starting to load model /home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct...
Loading safetensors checkpoint shards:   0% Completed | 0/30 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   3% Completed | 1/30 [00:00<00:13,  2.11it/s]
Loading safetensors checkpoint shards:   7% Completed | 2/30 [00:01<00:15,  1.85it/s]
Loading safetensors checkpoint shards:  10% Completed | 3/30 [00:01<00:14,  1.82it/s]
Loading safetensors checkpoint shards:  13% Completed | 4/30 [00:02<00:14,  1.81it/s]
Loading safetensors checkpoint shards:  17% Completed | 5/30 [00:02<00:13,  1.80it/s]
Loading safetensors checkpoint shards:  20% Completed | 6/30 [00:03<00:12,  1.97it/s]
Loading safetensors checkpoint shards:  23% Completed | 7/30 [00:03<00:11,  1.93it/s]
Loading safetensors checkpoint shards:  27% Completed | 8/30 [00:04<00:11,  1.89it/s]
Loading safetensors checkpoint shards:  30% Completed | 9/30 [00:04<00:11,  1.87it/s]
Loading safetensors checkpoint shards:  33% Completed | 10/30 [00:05<00:10,  1.92it/s]
Loading safetensors checkpoint shards:  37% Completed | 11/30 [00:05<00:10,  1.90it/s]
Loading safetensors checkpoint shards:  40% Completed | 12/30 [00:06<00:09,  1.94it/s]
Loading safetensors checkpoint shards:  43% Completed | 13/30 [00:06<00:08,  1.89it/s]
Loading safetensors checkpoint shards:  47% Completed | 14/30 [00:07<00:08,  1.87it/s]
Loading safetensors checkpoint shards:  50% Completed | 15/30 [00:07<00:08,  1.86it/s]
Loading safetensors checkpoint shards:  53% Completed | 16/30 [00:08<00:07,  1.85it/s]
Loading safetensors checkpoint shards:  57% Completed | 17/30 [00:08<00:06,  1.91it/s]
Loading safetensors checkpoint shards:  60% Completed | 18/30 [00:09<00:06,  1.89it/s]
Loading safetensors checkpoint shards:  63% Completed | 19/30 [00:10<00:05,  1.87it/s]
Loading safetensors checkpoint shards:  67% Completed | 20/30 [00:10<00:05,  1.86it/s]
Loading safetensors checkpoint shards:  70% Completed | 21/30 [00:11<00:04,  1.83it/s]
Loading safetensors checkpoint shards:  73% Completed | 22/30 [00:11<00:04,  1.83it/s]
Loading safetensors checkpoint shards:  77% Completed | 23/30 [00:12<00:03,  1.84it/s]
Loading safetensors checkpoint shards:  80% Completed | 24/30 [00:12<00:03,  1.84it/s]
Loading safetensors checkpoint shards:  83% Completed | 25/30 [00:13<00:02,  1.82it/s]
Loading safetensors checkpoint shards:  87% Completed | 26/30 [00:13<00:02,  1.82it/s]
Loading safetensors checkpoint shards:  90% Completed | 27/30 [00:14<00:01,  1.88it/s]
Loading safetensors checkpoint shards:  93% Completed | 28/30 [00:14<00:00,  2.38it/s]
(VllmWorkerProcess pid=479247) INFO 10-28 11:14:51 model_runner.py:1067] Loading model weights took 65.7407 GB
Loading safetensors checkpoint shards:  97% Completed | 29/30 [00:15<00:00,  2.24it/s]
Loading safetensors checkpoint shards: 100% Completed | 30/30 [00:15<00:00,  2.13it/s]
Loading safetensors checkpoint shards: 100% Completed | 30/30 [00:15<00:00,  1.92it/s]

INFO 10-28 11:14:52 model_runner.py:1067] Loading model weights took 65.7407 GB
INFO 10-28 11:14:52 distributed_gpu_executor.py:57] # GPU blocks: 0, # CPU blocks: 1638
INFO 10-28 11:14:52 distributed_gpu_executor.py:61] Maximum concurrency for 131072 tokens per request: 0.00x
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229] Exception in worker VllmWorkerProcess while processing method initialize_cache.
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229] Traceback (most recent call last):
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229]   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229]   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/worker.py", line 262, in initialize_cache
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229]   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/worker.py", line 487, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229]     raise ValueError("No available memory for the cache blocks. "
(VllmWorkerProcess pid=479247) ERROR 10-28 11:14:52 multiproc_worker_utils.py:229] ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/jovyan/.conda/envs/harness/bin/lm_eval", line 8, in <module>
[rank0]:     sys.exit(cli_evaluate())
[rank0]:              ^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/__main__.py", line 382, in cli_evaluate
[rank0]:     results = evaluator.simple_evaluate(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/utils.py", line 472, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/evaluator.py", line 204, in simple_evaluate
[rank0]:     lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
[rank0]:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/api/model.py", line 147, in create_from_arg_string
[rank0]:     return cls(**args, **args2)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/models/vllm_causallms.py", line 103, in __init__
[rank0]:     self.model = LLM(**self.model_args)
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 177, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 573, in from_engine_args
[rank0]:     engine = cls(
[rank0]:              ^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 348, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 496, in _initialize_kv_caches
[rank0]:     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/executor/distributed_gpu_executor.py", line 67, in initialize_cache
[rank0]:     self._run_workers("initialize_cache",
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/executor/multiproc_gpu_executor.py", line 192, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/worker.py", line 262, in initialize_cache
[rank0]:     raise_if_cache_size_invalid(num_gpu_blocks,
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/worker.py", line 487, in raise_if_cache_size_invalid
[rank0]:     raise ValueError("No available memory for the cache blocks. "
[rank0]: ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine.
INFO 10-28 11:14:53 multiproc_worker_utils.py:120] Killing local vLLM worker processes
[rank0]:[W1028 11:14:53.764496769 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/home/jovyan/.conda/envs/harness/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '`


**So, I changed the gpu_memory_utilization 0.80 -> 0.95 and max_model_len=4096. But, resource_tracker error also...**

`sh agieval.sh 
2024-10-28:11:17:39,546 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-28:11:17:45,519 INFO     [__main__.py:376] Selected Tasks: ['ai2_arc']
2024-10-28:11:17:45,523 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-28:11:17:45,523 INFO     [evaluator.py:201] Initializing vllm model, with arguments: {'pretrained': '/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct', 'tensor_parallel_size': 2, 'dtype': 'auto', 'gpu_memory_utilization': 0.95, 'max_model_len': 4096}
INFO 10-28 11:17:49 config.py:905] Defaulting to use mp for distributed inference
INFO 10-28 11:17:49 llm_engine.py:237] Initializing an LLM engine (v0.6.3.post1) with config: model='/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct', speculative_config=None, tokenizer='/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=/home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 10-28 11:17:49 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 56 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 10-28 11:17:49 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=481350) INFO 10-28 11:17:50 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=481350) INFO 10-28 11:17:50 utils.py:1008] Found nccl from library libnccl.so.2
INFO 10-28 11:17:50 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=481350) INFO 10-28 11:17:50 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 10-28 11:17:50 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=481350) INFO 10-28 11:17:50 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/jovyan/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 10-28 11:17:50 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/jovyan/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 10-28 11:17:50 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7f98ad287290>, local_subscribe_port=41799, remote_subscribe_port=None)
INFO 10-28 11:17:50 model_runner.py:1056] Starting to load model /home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct...
(VllmWorkerProcess pid=481350) INFO 10-28 11:17:50 model_runner.py:1056] Starting to load model /home/jovyan/data-vol-1/models/meta-llama__Llama3.1-70B-Instruct...
Loading safetensors checkpoint shards:   0% Completed | 0/30 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   3% Completed | 1/30 [00:00<00:13,  2.08it/s]
Loading safetensors checkpoint shards:   7% Completed | 2/30 [00:01<00:15,  1.85it/s]
Loading safetensors checkpoint shards:  10% Completed | 3/30 [00:01<00:14,  1.83it/s]
Loading safetensors checkpoint shards:  13% Completed | 4/30 [00:02<00:14,  1.81it/s]
Loading safetensors checkpoint shards:  17% Completed | 5/30 [00:02<00:13,  1.80it/s]
Loading safetensors checkpoint shards:  20% Completed | 6/30 [00:03<00:12,  1.99it/s]
Loading safetensors checkpoint shards:  23% Completed | 7/30 [00:03<00:11,  1.94it/s]
Loading safetensors checkpoint shards:  27% Completed | 8/30 [00:04<00:11,  1.91it/s]
Loading safetensors checkpoint shards:  30% Completed | 9/30 [00:04<00:11,  1.89it/s]
Loading safetensors checkpoint shards:  33% Completed | 10/30 [00:05<00:10,  1.94it/s]
Loading safetensors checkpoint shards:  37% Completed | 11/30 [00:05<00:09,  1.91it/s]
Loading safetensors checkpoint shards:  40% Completed | 12/30 [00:06<00:09,  1.96it/s]
Loading safetensors checkpoint shards:  43% Completed | 13/30 [00:06<00:08,  1.91it/s]
Loading safetensors checkpoint shards:  47% Completed | 14/30 [00:07<00:08,  1.92it/s]
Loading safetensors checkpoint shards:  50% Completed | 15/30 [00:07<00:07,  1.93it/s]
Loading safetensors checkpoint shards:  53% Completed | 16/30 [00:08<00:07,  1.94it/s]
Loading safetensors checkpoint shards:  57% Completed | 17/30 [00:08<00:06,  2.01it/s]
Loading safetensors checkpoint shards:  60% Completed | 18/30 [00:09<00:06,  1.99it/s]
Loading safetensors checkpoint shards:  63% Completed | 19/30 [00:09<00:05,  1.98it/s]
Loading safetensors checkpoint shards:  67% Completed | 20/30 [00:10<00:05,  1.97it/s]
Loading safetensors checkpoint shards:  70% Completed | 21/30 [00:10<00:04,  1.94it/s]
Loading safetensors checkpoint shards:  73% Completed | 22/30 [00:11<00:04,  1.95it/s]
Loading safetensors checkpoint shards:  77% Completed | 23/30 [00:11<00:03,  1.95it/s]
Loading safetensors checkpoint shards:  80% Completed | 24/30 [00:12<00:03,  1.96it/s]
Loading safetensors checkpoint shards:  83% Completed | 25/30 [00:12<00:02,  1.94it/s]
Loading safetensors checkpoint shards:  87% Completed | 26/30 [00:13<00:02,  1.94it/s]
Loading safetensors checkpoint shards:  90% Completed | 27/30 [00:13<00:01,  2.01it/s]
Loading safetensors checkpoint shards:  93% Completed | 28/30 [00:14<00:00,  2.54it/s]
Loading safetensors checkpoint shards:  97% Completed | 29/30 [00:14<00:00,  2.37it/s]
Loading safetensors checkpoint shards: 100% Completed | 30/30 [00:15<00:00,  2.22it/s]
Loading safetensors checkpoint shards: 100% Completed | 30/30 [00:15<00:00,  1.99it/s]

INFO 10-28 11:18:06 model_runner.py:1067] Loading model weights took 65.7407 GB
(VllmWorkerProcess pid=481350) INFO 10-28 11:18:06 model_runner.py:1067] Loading model weights took 65.7407 GB
INFO 10-28 11:18:08 distributed_gpu_executor.py:57] # GPU blocks: 3140, # CPU blocks: 1638
INFO 10-28 11:18:08 distributed_gpu_executor.py:61] Maximum concurrency for 4096 tokens per request: 12.27x
(VllmWorkerProcess pid=481350) INFO 10-28 11:18:10 model_runner.py:1395] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=481350) INFO 10-28 11:18:10 model_runner.py:1399] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 10-28 11:18:10 model_runner.py:1395] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 10-28 11:18:10 model_runner.py:1399] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=481350) INFO 10-28 11:18:27 custom_all_reduce.py:233] Registering 5635 cuda graph addresses
INFO 10-28 11:18:27 custom_all_reduce.py:233] Registering 5635 cuda graph addresses
INFO 10-28 11:18:27 model_runner.py:1523] Graph capturing finished in 17 secs.
(VllmWorkerProcess pid=481350) INFO 10-28 11:18:27 model_runner.py:1523] Graph capturing finished in 17 secs.
2024-10-28:11:18:28,475 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
2024-10-28:11:18:28,475 INFO     [task.py:422] Building contexts for arc_easy on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2376/2376 [00:01<00:00, 1825.79it/s]
2024-10-28:11:18:29,879 INFO     [task.py:422] Building contexts for arc_challenge on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1172/1172 [00:00<00:00, 1842.83it/s]
2024-10-28:11:18:30,565 INFO     [evaluator.py:489] Running loglikelihood requests
Running loglikelihood requests:   0%|                                                                                                                                                                | 0/14188 [00:00<?, ?it/sINFO 10-28 11:18:35 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241028-111835.pkl...            | 0/14188 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
WARNING 10-28 11:18:35 model_runner_base.py:143] Failed to pickle inputs of failed execution: Can't pickle local object 'weak_bind.<locals>.weak_bound'
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1701, in execute_model
[rank0]:     output: SamplerOutput = self.model.sample(
[rank0]:                             ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 573, in sample
[rank0]:     next_tokens = self.sampler(logits, sampling_metadata)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 260, in forward
[rank0]:     logits = logits.to(torch.float)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.96 GiB. GPU 0 has a total capacity of 79.33 GiB of which 1.62 GiB is free. Process 2460811 has 77.69 GiB memory in use. Of the allocated memory 74.59 GiB is allocated by PyTorch, with 36.08 MiB allocated in private pools (e.g., CUDA Graphs), and 174.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/jovyan/.conda/envs/harness/bin/lm_eval", line 8, in <module>
[rank0]:     sys.exit(cli_evaluate())
[rank0]:              ^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/__main__.py", line 382, in cli_evaluate
[rank0]:     results = evaluator.simple_evaluate(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/utils.py", line 472, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/evaluator.py", line 301, in simple_evaluate
[rank0]:     results = evaluate(
[rank0]:               ^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/utils.py", line 472, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/evaluator.py", line 500, in evaluate
[rank0]:     resps = getattr(lm, reqtype)(cloned_reqs)
[rank0]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/api/model.py", line 378, in loglikelihood
[rank0]:     return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/models/vllm_causallms.py", line 442, in _loglikelihood_tokens
[rank0]:     outputs = self._model_generate(requests=inputs, generate=False)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/workspace/platform/lm-evaluation-harness-main/lm_eval/models/vllm_causallms.py", line 268, in _model_generate
[rank0]:     outputs = self.model.generate(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/utils.py", line 1063, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 353, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 879, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 1389, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/executor/distributed_gpu_executor.py", line 82, in execute_model
[rank0]:     driver_outputs = self._driver_execute_model(execute_model_req)
[rank0]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/executor/multiproc_gpu_executor.py", line 155, in _driver_execute_model
[rank0]:     return self.driver_worker.execute_model(execute_model_req)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 327, in execute_model
[rank0]:     output = self.model_runner.execute_model(
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/jovyan/.conda/envs/harness/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
[rank0]:     raise type(err)(f"Error in model execution: "
[rank0]: torch.OutOfMemoryError: Error in model execution: CUDA out of memory. Tried to allocate 1.96 GiB. GPU 0 has a total capacity of 79.33 GiB of which 1.62 GiB is free. Process 2460811 has 77.69 GiB memory in use. Of the allocated memory 74.59 GiB is allocated by PyTorch, with 36.08 MiB allocated in private pools (e.g., CUDA Graphs), and 174.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
INFO 10-28 11:18:36 multiproc_worker_utils.py:120] Killing local vLLM worker processes
Processed prompts:   0%|                                                                                                                         | 0/14188 [00:03<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Running loglikelihood requests:   0%|                                                                                                                                                                | 0/14188 [00:04<?, ?it/s]
[rank0]:[W1028 11:18:37.518066433 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/home/jovyan/.conda/envs/harness/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '`


please help me.



I'm encountering the same problem.

I'm using the following code to evaluate LLaMA 3.1 8B, and it works:

```bash
lm_eval --model vllm --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,tensor_parallel_size=4,dtype=float16,gpu_memory_utilization=0.8 --tasks multimedqa,tmmluplus,toxigen --batch_size auto --output_path /mnt/out_disk/llm_eval_result/meta_llama31_8b_instruct --trust_remote_code
```

However, when I try to evaluate Phi-3 small or Qwen 2.5 7B using the following code, I get this problem:

```bash
lm_eval --model vllm --model_args pretrained=microsoft/Phi-3-small-128k-instruct,tensor_parallel_size=4,dtype=float16,gpu_memory_utilization=0.8 --tasks multimedqa,tmmluplus,toxigen --batch_size auto --output_path /mnt/out_disk/llm_eval_result/ms_phi3_7b_instruct --trust_remote_code
```

```bash
lm_eval --model vllm --model_args pretrained=Qwen/Qwen2.5-7B-Instruct,tensor_parallel_size=4,dtype=float16,gpu_memory_utilization=0.8 --tasks multimedqa,tmmluplus,toxigen --batch_size auto --output_path /mnt/out_disk/llm_eval_result/qwen25_7b_instruct --trust_remote_code
```



Can you share an info about the environment contains gpu?
I think it is not caused by multiprocessing but oom or something.