Optional resident mode for the quantized tables (VLLM_PLE_QUANT_RESIDENT)
Adds an opt-in flag that copies the quantized PLE shards into anonymous host RAM at load instead of serving them mmapped. Unset, behaviour is byte-identical to today. Rebased on current main, so the recent Parameter-stub and global-scale fixes are untouched.
Why: the mmap design is right for the hosts you measured. Page cache is reclaimable, and it is what lets a low-RAM machine run this model at all. On a host with a slow root device it inverts. Ours is an OCZ Agility 3 SATA SSD at 486 MB/s sequential, so any eviction puts PLE gathers back on that disk mid-decode. File-backed mappings also get almost no huge pages on this kernel, FileHugePages 126 MiB against AnonHugePages 36 GiB, so the 49 GiB FP8 table is walked through roughly 12.8M 4 KiB PTEs.
Three details in the patch:
- preflight against actually available RAM, so an undersized host fails in seconds with a clear message instead of OOM-ing minutes into a load
clone()per shard during load, never a pageable table converted afterwards, so the transient cost is one shard (about 410 MiB) rather than a second full tableposix_fadvise(DONTNEED)after each shard is copied, so the cgroup does not account both the anonymous copy and the file pages it came from
Deliberately not pinned. Your CPU worker dequantizes into buffers that are already shared and CUDA-registered, so pinning buys no DMA benefit while making 49 GiB permanently page-locked. Plain anonymous memory plus memory.swap.max=0 gives the same no-disk guarantee and keeps THP.
Evidence, hedged where it should be. This is a guarantee argument more than a benchmark:
- cold pages caused severe first-request latency, and a sequential prewarm of the sidecar materially improved decode
- the prewarm has to run after API readiness, because model loading evicts an earlier one
- with resident mode the table stayed 100 percent resident across a 50-request soak, zero errors, zero silent outputs, zero VRAM growth
- we did not run a controlled resident vs warm-mmap A/B, so no steady-state throughput claim
Running in production since 1 September on 4x RTX 3090, FP8 tables, TP4, 262,144 context, serving a W4A16 body.
The approach follows the make_resident() in jieen1/BlackweLLM, which solved this first. Ours differs in the two points above (per shard, not pinned).
One known rough edge: the preflight uses total_rows * width as the byte estimate, which is exact for ples_fp8 but roughly 2x conservative for ples_int4, so an int4 user on a tight host could be refused unnecessarily. Happy to make it layout-aware, I left it conservative because our production path is FP8 and I did not want to ship an int4 estimate I have not exercised.
Read the diff line by line. The design is right, and unset it really is a no-op.
Your reasoning about not pinning matches the code, for anyone reading later: the offload worker's output staging buffers are already pin_memory=True (_pinned_bufs in worker_image_quant.py), and the table itself is only touched CPU-side during dequant, so page-locking 49 GiB would buy no DMA path that isn't already pinned.
Two things to fix before we merge, both in the preflight.
1. SC_AVPHYS_PAGES is MemFree, not available memory, and it isn't cgroup-aware. It fails in both directions. Restart the server after a run that mmapped the sidecar and the table's pages are sitting in page cache: MemFree is small, MemAvailable is large, and the preflight refuses a copy that would have succeeded. That's the ordinary case, not a corner. In the other direction, the serve path in our README is docker run, where sysconf reports the host's free pages and ignores --memory; the preflight passes and the cgroup kills the copy part-way through. Suggest MemAvailable from /proc/meminfo, and where a cgroup limit exists, the min of that and memory.max - memory.current (v2) or limit_in_bytes - usage_in_bytes (v1).
2. total_rows * width is off for all three layouts, fp8 included. Measured against what's in the repo:
| sidecar | actual resident | your estimate | |
|---|---|---|---|
ples_fp8 |
48.87 GiB (51.2 GB weights + 1.28 GB per-row fp32 scales) | 47.68 GiB | under by 1.19 GiB |
ples_int4 |
29.80 GiB | 47.68 GiB | 1.60x over |
ples_nvfp4 |
26.82 GiB | 47.68 GiB | 1.78x over |
The fp8 miss is small, but it's the layout you run and it's in the unsafe direction — the per-row scales aren't in the estimate. sum(os.path.getsize(shard)) is exact for every layout and needs no layout table, which also disposes of the int4 rough edge you flagged.
Nit: libc.posix_fadvise returns the errno rather than setting it, and ctypes won't raise, so the except OSError around it can't fire. Check the return value or drop the handler.
On evidence, we don't need the A/B. Our numbers already say the page cache does this job when there's RAM to spare: the INT4 sidecar with a warm cache holds 129.6 tok/s under MTP against 142.6 for a fully in-RAM BF16 table, while a disk-backed table under the same speculation collapses to 77–82. So the flag buys a residency guarantee rather than throughput, which is roughly how you described it, and it's how we'd document it. A slow root device and no THP on file-backed pages is a good enough reason for the flag to exist.
We want to boot it here before merging — we have a rule about not publishing an overlay we haven't run, learned the hard way. Once it's in, the flag goes in the README with the preflight caveat stated.
Thanks for the posix_fadvise step in particular. Dropping the file pages after each copy is the detail most implementations skip, and it's the one that keeps the cgroup accounting honest.
Booted it. One RTX PRO 6000, 176 GB host, INT4 sidecar, mixed checkpoint, five configurations, page cache dropped before each arm.
Unset is a true no-op: same log line, KV pool identical to the token (426,639 without MTP, 134,192 with), 445 s vs 449 s to healthy. Copying 29.8 GiB costs 4–11 s of boot.
Your THP argument is right on this kernel. FileHugePages reads 0.0 GiB in every arm; the resident copy shows anon_thp 32.0 GiB. Not a hypothesis here, just true.
Steady state is close to a tie, and the warm-up is the whole story:
| first request | warm asymptote | reqs to within 5% | |
|---|---|---|---|
| mmapped | 19.0 tok/s | 93.0 | ~150 |
| resident | 74.3 | 96.8 | ~3 |
| mmapped + MTP | 12.6 | 125.4 | ~125 |
| resident + MTP | 86.5 | 128.2 | ~25 |
That's +4.1% and +2.2% warm, the second inside the run-to-run spread. But the mmapped table starts at 19 tok/s and needs roughly 150 requests to converge, because loading the checkpoint evicts the sidecar from page cache — your observation about prewarm having to run after API readiness, seen from the other side. Speculation makes the cold case worse, not better (12.6 tok/s on the first request). So I'd document the flag as a cold-start and restart-frequency feature rather than a throughput one, which also matches how carefully you hedged the evidence.
Both preflight problems are now measured rather than argued. With the page cache deliberately filled:
MemoryError: PLE resident mode needs about 47.7 GiB for the table but only 1.9 GiB
of host RAM is available; unset VLLM_PLE_QUANT_RESIDENT to serve it mmapped
MemAvailable at that instant was 173.7 GiB. The host had three and a half times what it needed and the check refused. And the announced 47.7 GiB against a cgroup anon delta of exactly 29.8 GiB, measured twice — the INT4 sidecar's true size.
Fix those two and we'll merge. Happy to push the change myself if you'd rather not.
Pushed the two preflight fixes onto this branch rather than leave it sitting, since the flag itself is sound and default-off. Both are validated on one RTX PRO 6000 with a 176 GiB host, before and after.
The metric. SC_AVPHYS_PAGES is MemFree, which fails in both directions, and I could show each on the same box:
| old metric | new metric | truth | |
|---|---|---|---|
| host, page cache warm | 1.4 GiB | 168.8 GiB | MemAvailable 173.7 GiB |
inside --memory=24g |
71.45 GiB | 23.99 GiB | a 24 GiB cage |
The second row is the one that would have hurt: the old check reads the host's free pages, so it waves a 29.8 GiB copy into a 24 GiB container and leaves the OOM killer to sort it out. It now reads MemAvailable, and in a container the cgroup limit less its reclaimable file pages. That last part matters more than it looks — memory.current counts the checkpoint we just streamed, so a plain limit - current would refuse inside a container for exactly the same wrong reason the original refused on the host.
Capped at 24g it now refuses cleanly, OOMKilled=false, which is the part I wanted to see:
MemoryError: PLE resident mode needs about 29.8 GiB for the table but only
19.2 GiB is available to this process; unset VLLM_PLE_QUANT_RESIDENT to
serve it mmapped
The size. Summing the shard files: 48.88 / 29.80 / 26.82 GiB for fp8 / int4 / nvfp4. The int4 figure is exactly the anon delta the cgroup measured during the boot. Your instinct that fp8 was "exact" was close but it's actually 1.19 GiB short, since the per-row scales aren't in rows*width — the unsafe direction, on the layout you run.
One change beyond what I flagged. A unit case caught that if neither /proc/meminfo nor sysconf can be read, the preflight would raise ValueError and block the boot. It now returns None and the check is skipped with "headroom unknown" in the log. Failing fast is a convenience; it shouldn't be able to stop a server from starting. Also made the posix_fadvise error path real.
Three boots, all green: unset is still a no-op (128 shards mmapped, KV 426,639, identical to before); starved page cache now boots and reports the right size (copying about 29.8 GiB into anonymous RAM (168.8 GiB available)); capped container refuses as above.
Merging. README documents the flag as a cold-start feature with the measured numbers and credits you. Thanks for the patch and for hedging the evidence honestly — the posix_fadvise step in particular is the detail most implementations skip.
Preflight fixes validated on the box; merging.