Repository: mlcommons/inference
Issue: Llama 3.1 8B Server Scenario vLLM Shutting down
Root cause: Algorithmic Error

When running the following command:
```bash
OUTPUT_LOG_DIR=offline-accuracy-logs

python -u main.py --scenario Offline \
                --model-path ${CHECKPOINT_PATH} \
                --batch-size 16 \
                --accuracy \
                --dtype float16 \
                --user-conf user.conf \
                --total-sample-count 13368 \
                --dataset-path ${DATASET_FOLDER} \
                --output-log-dir ${OUTPUT_LOG_DIR} \
                --tensor-parallel-size 8 \
                --vllm


ACCURACY_LOG_FILE=${OUTPUT_LOG_DIR}/mlperf_log_accuracy.json
if [ -e ${ACCURACY_LOG_FILE} ]; then
        python evaluation.py --mlperf-accuracy-file ${ACCURACY_LOG_FILE} \
                --dataset-file ${DATASET_FOLDER} --dtype int32
fi
 ```
And the equivalent one for the Server Performance benchmark with the default [user.conf](https://github.com/mlcommons/inference/blob/master/language/llama3.1-8b/user.conf) vLLM server gets shut down after two request:

```bash
OUTPUT_LOG_DIR=server-accuracy-logs

python -u main.py --scenario Server \
                --model-path ${CHECKPOINT_PATH} \
                --batch-size 16 \
                --accuracy \
                --dtype float16 \
                --user-conf user.conf \
                --total-sample-count 13368 \
                --dataset-path ${DATASET_FOLDER} \
                --output-log-dir ${OUTPUT_LOG_DIR} \
                --tensor-parallel-size 8 \
                --vllm

ACCURACY_LOG_FILE=${OUTPUT_LOG_DIR}/mlperf_log_accuracy.json
if [ -e ${ACCURACY_LOG_FILE} ]; then
        python evaluation.py --mlperf-accuracy-file ${ACCURACY_LOG_FILE} \
                --dataset-file ${DATASET_FOLDER} --dtype int32
fi
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
INFO:Llama-8B-Dataset:Loading dataset...
INFO:Llama-8B-Dataset:Finished loading dataset.
INFO:Llama-8B-SUT:Loading model
WARNING 07-18 12:25:54 config.py:1674] Casting torch.bfloat16 to torch.float16.
INFO 07-18 12:25:59 config.py:887] Defaulting to use mp for distributed inference
WARNING 07-18 12:25:59 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
INFO 07-18 12:25:59 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
INFO 07-18 12:25:59 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='meta-llama/Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=meta-llama/Llama-3.1-8B-Instruct, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 07-18 12:25:59 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 192 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 07-18 12:25:59 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/mnt/co-research/home/rodri/miniconda3/envs/mlcflow/lib/python3.12/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:07 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:07 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:08 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:08 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:08 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:08 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:08 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:11 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:11 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:14 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /mnt/co-research/home/rodri/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
INFO 07-18 12:26:14 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7ff1112f8f20>, local_subscribe_port=58153, remote_subscribe_port=None)
INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:14 model_runner.py:1060] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:03<00:11,  3.83s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:07<00:07,  3.85s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:08<00:02,  2.35s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:11<00:00,  2.80s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:11<00:00,  2.94s/it]

(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:26 model_runner.py:1071] Loading model weights took 1.9028 GB
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:26 model_runner.py:1071] Loading model weights took 1.9028 GB
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:26 model_runner.py:1071] Loading model weights took 1.9028 GB
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:26 model_runner.py:1071] Loading model weights took 1.9028 GB
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:26 model_runner.py:1071] Loading model weights took 1.9028 GB
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:26 model_runner.py:1071] Loading model weights took 1.9028 GB
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:26 model_runner.py:1071] Loading model weights took 1.9028 GB
INFO 07-18 12:26:27 model_runner.py:1071] Loading model weights took 1.9028 GB
INFO 07-18 12:26:29 distributed_gpu_executor.py:57] # GPU blocks: 464716, # CPU blocks: 16384
INFO 07-18 12:26:29 distributed_gpu_executor.py:61] Maximum concurrency for 131072 tokens per request: 56.73x
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:31 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:31 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:31 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:31 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:32 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 07-18 12:26:32 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:32 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 07-18 12:26:32 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:32 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:32 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:32 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:32 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:32 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:32 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:32 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:32 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:46 custom_all_reduce.py:233] Registering 2275 cuda graph addresses
(VllmWorkerProcess pid=3311319) INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
(VllmWorkerProcess pid=3311316) INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
(VllmWorkerProcess pid=3311321) INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
(VllmWorkerProcess pid=3311317) INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
(VllmWorkerProcess pid=3311318) INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
(VllmWorkerProcess pid=3311320) INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
(VllmWorkerProcess pid=3311322) INFO 07-18 12:26:46 model_runner.py:1530] Graph capturing finished in 14 secs.
INFO:Llama-8B-SUT:Loaded model
INFO:Llama-8B-MAIN:Starting Benchmark run
INFO 07-18 12:26:50 async_llm_engine.py:209] Added request 0.
INFO 07-18 12:26:50 async_llm_engine.py:177] Finished request 0.
INFO 07-18 12:26:50 async_llm_engine.py:63] Engine is gracefully shutting down.
INFO 07-18 12:26:50 async_llm_engine.py:209] Added request 1.
INFO 07-18 12:26:50 async_llm_engine.py:63] Engine is gracefully shutting down.
```

Working with Local Environment Run configuration without docker.

We discarded this scenario for v5.1 inference submission