vLLM 0.23.1rc1.dev774+g2b802ad5b.d20260706. . load this model encounter OOM !!
Environment
Field Value
vLLM version 0.23.1rc1.dev774+g2b802ad5b.d20260706 (self-compiled from vllm-next)
GPU NVIDIA RTX PRO 6000 Blackwell, 96 GB VRAM, sm_120
CUDA 12.x
Model Qwen3.6-27B-NVFP4
Date 2026-07-07
Loading Qwen3.6-27B-NVFP4 with vLLM v0.23.1rc1 dev consistently crashes with an OOM during Stage 0 (model weight loading), even with generous GPU memory fraction (--gpu-memory-utilization 0.65). The same hardware loads the FP8 variant (Qwen3.6-27B-FP8) without issue in ~121 seconds.
Multiple attempts were made over two days (2026-07-06 and 07-07), all reproducing the same failure:
Date/Time Model gpu_mem Result
07/06 21:29 Qwen3.6-27B-NVFP4 0.65 OOM (Stage 0)
07/06 21:38 Qwen3.6-27B-NVFP4 0.65 OOM (Stage 0)
07/06 22:55 Qwen3.6-27B-NVFP4 0.65 OOM (Stage 0)
07/06 22:55 Qwen3.6-27B-NVFP4 0.65 OOM (Stage 0)
07/07 Qwen3.6-27B-FP8 0.65 OK (121s)
True error log:
" text
[ 9794.850361] Out of memory: Killed process 108229 (vLLM::EngineCor) total-vm:249406116kB, anon-rss:1175116kB, file-rss:57080kB, shmem-rss:258048kB, UID:1000 pgtables:9336kB oom_score_adj:200 "
