Set expandable_segments in the 2 x DGX Spark environment
Adds PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" to the environment that both DGX Spark nodes share.
In two attempts at this section's manual launch on two GB10 machines (128 GB each), the second with the KV cache fixed as below, each host's available memory fell by 17 to 19 GiB within about 10 seconds after Loading weights took, before the server came up. With this setting on both nodes the same start completed and served (MTP 5, answered and called tools). The full report is in discussion #7.
| Both nodes | Lowest available memory (two hosts) | Result |
|---|---|---|
| Without the setting | 7.8, 7.9 GiB, 5 to 7 s after the weights loaded | stopped by our 8 GiB floor |
| With it | 13.7, 14.5 GiB during the start; 13.9, 15.1 GiB while serving | served |
Both rows used --gpu-memory-utilization 0.80 (0.87 here), --max-model-len 131072 (500000) and --kv-cache-memory-bytes 7650410496, with the weights at 76348ddb read offline from the pinned snapshot and eugr/spark-vllm-b12x@sha256:03a95654…. Apart from serving on 127.0.0.1 and adding VLLM_HOST_IP and NCCL_IB_GID_INDEX=3 for our direct link, the rest followed this section. The 8 GiB floor is a watchdog of ours; vLLM had logged nothing after Loading weights took when the watchdog stopped the start. We have not run this section's own values, with or without the setting.
The RTX PRO 6000 section already sets expandable_segments:True,large_segment_size_mb:12; we ran only expandable_segments:True. The one-step launcher's recipe does not set it either: https://github.com/eugr/spark-vllm-docker/issues/441
Written with Weave (My Symbiotic AI)