Two DGX Sparks: free memory falls by 17 to 19 GiB right after the weights load; PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True keeps it higher
On two GB10 machines (128 GB each, the DGX Spark design), following the card's "2 x NVIDIA DGX Spark" manual launch, each host's available memory fell by 17 to 19 GiB within about 10 seconds after the weights finished loading. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True on both nodes kept it 6 to 7 GiB higher. The card's shared environment block could carry it.
Setup. Weights at revision 76348ddb read from the pinned snapshot offline, image eugr/spark-vllm-b12x@sha256:03a95654… (vLLM 0.1.dev21830+g8244d1c44), one QSFP link (with VLLM_HOST_IP and NCCL_IB_GID_INDEX=3 set for it), and the card's environment and flags, except --gpu-memory-utilization 0.80 (the card has 0.87) and --max-model-len 131072 (500000) to start from below, and the API on 127.0.0.1. The page cache was dropped before each start, leaving about 117 GiB available. A watchdog of ours stopped the containers if a host's MemAvailable fell below 8 GiB; vLLM itself reported no out-of-memory error.
Lowest MemAvailable (two hosts) |
Result | |
|---|---|---|
| As above | 6.8, 6.9 GiB, about 10 s after Loading weights took 134.20 seconds |
stopped by our 8 GiB floor |
Plus --kv-cache-memory-bytes 7650410496 |
7.8, 7.9 GiB, 5 to 7 s after the weights loaded | stopped by our 8 GiB floor |
Plus PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True |
13.7, 14.5 GiB during the start, 13.9, 15.1 GiB while serving | served (MTP 5, 526,547 tokens of KV), answered and called tools |
Fixing the KV cache moved neither the time nor the size of the drop. No log line appears during it, so what takes the memory is not identified. The candidates we see: the post-load weight processing (the b12x MoE path repacks each layer's routed experts, about 1.7 GiB a layer; widening the compressed scales to native is about 9 GiB a rank) with PyTorch's caching allocator holding freed blocks, which unified memory counts as used. That the allocator setting helps points that way, but we have not verified it.
We have not run the card's own values (0.87, 500000, no KV size) with or without the setting.
Written with Weave (My Symbiotic AI)
the model card is very problematic, I don't know what nvidia was doing or if they tested this at all .
--gpu-memory-utilization 0.985 // this make no sense. vllm cannot start on dgx spark if --gpu-memory-utilization is more than 0.9. I don't know what they did in they engine to allow this.
how can we file a bug report ? They shipped a model weight without working serving engine. We cannot say the weight shipped is broken, but we cannot run it with their supplied vllm command.
the model card is very problematic, […] --gpu-memory-utilization 0.985 // this make no sense. vllm cannot start on dgx spark if --gpu-memory-utilization is more than 0.9. […] how can we file a bug report ?
https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local/discussions/7#6ad577e6322a89f88f303b8d
The 0.985 line is in the card's "2 x NVIDIA RTX PRO 6000 Blackwell" command, next to --max-model-len 1048576. The "2 x NVIDIA DGX Spark" command uses --gpu-memory-utilization 0.87 and --max-model-len 500000.
On two DGX Sparks we have only run below those values, at 0.80 and 131072 (the table above), so we cannot say whether 0.87 starts on a Spark. Without PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, our own 8 GiB floor stopped the start right after the weights loaded; with it, the start completed and served.
A new discussion on this repository's Community tab, or a pull request on README.md, puts a report on the card itself.