GLM-5.3-NVFP4-Spark won’t load on 4x DGX Spark TP=4 — NV_ERR_NO_MEMORY
#1
by dejan - opened
Setup:
- 4x DGX Spark GB10, 128GB unified (~117 GiB usable), Ubuntu, MTU 9000 RoCE fabric
- eugr/spark-vllm-b12x:verified-c3b44d3f, vLLM 0.1.dev20489+ga50ebee1d, B12X backends
- local-inference-lab/GLM-5.3-NVFP4-Spark, 755B MoE, 78+1 layers, 405.6 GiB total / ~101.4 GiB per TP rank
- TP=4 over NCCL NET/IB, InstantTensor buffered 64M/8M/1/3, copy:false
Minimal repro config:
- modelopt_mixed, B12X attn/linear/MoE, FP8 KV, block 256
- MTP off, max-model-len 8192, max-num-seqs 1, max-num-batched-tokens 4096, kv-cache-memory-bytes 2G, enforce-eager
- --cpu-offload-gb 48 --offload-backend uva, RoCE all-reduce 2M
Issue:
- All ranks die during Loading model from scratch with kernel NVRM: nvCheckOkFailedNoLog: Out of memory [NV_ERR_NO_MEMORY] from _memdescAllocInternal
- Exit 137/255, OOMKilled=false, hosts wedge until manual power-cycle
- Happens with 117 GiB free pre-launch, page cache dropped, 108G container cap
- Same failure at offload 12/24/48G, with and without MTP, 32K and 8K ctx