nktlabs/dsv41-flash-engram-disk-prestage

Engram off NVMe: running DeepSeek-V4.1-Flash on a 125 GB host

A patch set and recipe that lets diffbot/DeepSeek-V4.1-Flash-EXL3-2.0bpw-2x-RTX-PRO-6000 serve at 80.2 tok/s single-stream (449.7 tok/s aggregate at 8 streams, GSM8K-200 97.5%) on a machine with 125 GB of host RAM and 18 GB of it in use — instead of the ~280 GB the pinned-Engram path needs.

No weights here. This is four patched files, a serve script, and the measurements that justify them.

Built for diffbot/DeepSeek-V4.1-Flash-EXL3-2.0bpw-2x-RTX-PRO-6000, which quantises deepseek-ai/DeepSeek-V4.1-Flash. Those are linked here in prose rather than via base_model: metadata on purpose: the Hub's only relation values are adapter, merge, quantized and finetune, and with none set it infers finetune — this repo is none of those. It is a serving recipe, and claiming a fine-tune of someone else's model to buy a tree link would be a lie.

The short version

DeepSeek-V4.1-Flash needs 48 random 256-byte Engram rows per token out of a 190 GiB table. The reference path pins that table in host RAM. The vendor's on-disk fallback (DSV41_ENGRAM_DISK=1) works but reads rows inside the forward pass, which forces eager mode and costs ~3× decode throughput (26.1 tok/s measured here; 31.5 reported by diffbot).

The read does not need to be there. DeepseekV41ModelState.prepare_inputs already runs eagerly, immediately before run_fullgraph. Moving the Engram gather into that phase — hash kernel early, preadv into a pinned buffer, one H2D copy into a persistent GPU tensor — lets CUDA graphs stay on, because nothing inside the captured region touches a file.

Configuration	1 stream	4 streams (agg)	8 streams (agg)	Host RAM
Pinned host RAM (diffbot, reported)	113.3 tok/s	277.2 tok/s	—	~280 GB
Disk + eager (ENGRAM_DISK=1)	26.1 tok/s	107.6 tok/s	—	12 GB RSS
Disk + prestage + graphs	80.2 tok/s	229.2 tok/s	449.7 tok/s	18 GB RSS

Prefill 1719 tok/s single / 3324 aggregate. Per-stream decode holds ~56–57 tok/s from 4 to 8 streams — linear through max_num_seqs, so the NVMe row count is not the cap. GSM8K-200 = 195/200 = 97.5%, matching the pack's published thinking-off figure: prestaging changes when rows are fetched, not which.

What's in this repo

patches/prestage/engram.py         # DiskEngramTable.prestage(), check_prestaged(), disk_gather_local()
patches/prestage/engram_disk.py    # row/scale layout helper used by the gather
patches/prestage/model.py          # DeepseekV4Model.engram_prestage(), _check_engram_prestage(), step timer
patches/prestage/model_state.py    # prepare_inputs hook + VL-wrapper model resolver
serve/serve-prestage.sh            # diffbot's serve.sh + the prestage mounts; no rebuild
PATCH-SUMMARY.md                   # every changed symbol, and how to regenerate the diffs yourself
NOTICE.md                          # attribution and licences (vLLM Apache-2.0, pack MIT)

This is an overlay on the pack's own recipe/ tree, not a standalone runner. The layout above deliberately mirrors recipe/: copy patches/prestage/ and serve/serve-prestage.sh into your checkout of it. serve-prestage.sh is diffbot's recipe/serve/serve.sh with the prestage mounts added, and it still needs three things from their tree that are not redistributed here:

• patches/vllm/sitecustomize.py — always mounted
• patches/flashinfer/sparse_mla_sm120_prefill.cu and patches/flashinfer/empty-aot — mounted unless you pass FI_PATCH=
• kernels/build/xmoe — required unless you pass XMOE_EXT= for the stock kernel

The four Python files are whole-file replacements because that is how they are consumed: the serve script bind-mounts them over the image, so there is no rebuild. PATCH-SUMMARY.md lists exactly what changed and gives the one-liner to diff them against your own image's source, which is the right way to review this and the right way to re-apply it on a newer image tag.

Quick start

# 0. The pack, which carries the recipe/ tree this overlays.
hf download diffbot/DeepSeek-V4.1-Flash-EXL3-2.0bpw-2x-RTX-PRO-6000 --local-dir /models/dsv41

# 1. Base image. Has the merged V4.1-Flash model code; the v0.29.0 tag does NOT.
docker pull vllm/vllm-openai:deepseekv41-flash-0909

# 2. sm_120 overlay and the xmoe kernel, from the pack's own recipe.
cd /models/dsv41/recipe && ./docker/build.sh && ./kernels/build.sh

# 3. Overlay this repo onto that tree.
cp -r patches/prestage        /models/dsv41/recipe/patches/
cp    serve/serve-prestage.sh /models/dsv41/recipe/serve/

# 4. PARITY FIRST. Eager-only; does the real lookup as well and asserts the
#    prestaged rows are byte-identical. Do not skip this.
cd /models/dsv41/recipe
API_KEY=hunter2 ENGRAM_DISK=1 ENGRAM_PRESTAGE=1 ENGRAM_PRESTAGE_CHECK=1 EAGER=1 \
  ./serve/serve-prestage.sh

# 5. Then graphs on, for the numbers in the table above.
API_KEY=hunter2 ENGRAM_DISK=1 ENGRAM_PRESTAGE=1 EAGER=0 \
  ./serve/serve-prestage.sh

API_KEY is required — the script exits without it. ENGRAM_DISK, ENGRAM_PRESTAGE, ENGRAM_PRESTAGE_CHECK and EAGER are script-level knobs; the script translates them into the container's DSV41_ENGRAM_* environment, so those are the names you will see inside the container and in the patched code. PORT defaults to 8000, PACK to $MODELS_DIR/DeepSeek-V4.1-Flash-EXL3-2.0bpw, MAX_NUM_SEQS to 8, and THINKING to default — every number below was measured with THINKING=off.

Requirements: 2× 96 GB sm_120 cards; ~160 GB free on a real NVMe for shards 1–46; another 203 GB for Engram shards 47–48 (reflink them if your filesystem can — ours came from an existing FP8 pack unchanged); ~20 GB host RAM. Model load takes 159 s and both GPUs.

Why this works: measure before you design

The instinct is to treat 48 random reads per token as a storage problem and go shopping. We measured first — 32 threads issuing os.preadv of 4 KiB rows against shards 47/48, with the GPUs busy serving something else, so the numbers include real interference:

Pattern	p50	p99	max
48 rows, O_DIRECT, cold	1.41 ms	3.87 ms	247 ms
48 rows, buffered, first pass	1.89 ms	4.99 ms	—
48 rows, warm page cache	0.87 ms	2.61 ms	—
288 rows (k=5 lookback shape), cold	5.26 ms	21.5 ms	—

O_DIRECT beat the buffered first pass: the copy into page cache costs more than it saves on a cold read. And the eager-mode throughput confirmed the diagnosis from the other side — 26.1 tok/s at one stream, 107.6 aggregate at four, exactly linear. Nothing was saturating.

This is a scheduling problem wearing a storage problem's clothes. The fix is to move the read out of the graphed region, not to make the device faster.

The prestage, per decode step

1. prepare_inputs calls engram_prestage() on the model, which runs the hash kernel eagerly — the same kernel the forward would have run — to learn which rows this step needs. The V2 runner has use_slot_cache=False, so it is fed dummy slot/block tensors.
2. One .cpu() sync brings the hash ids to the host.
3. Engram.prestage() gathers those rows with preadv into a pinned host buffer, event-guarded so it cannot be overwritten while a prior copy is in flight, then issues one H2D copy into a persistent GPU tensor (staged_rows).
4. The graphed forward reads prestaged_hashes[:T] and skips the lookup entirely. It raises if prestaged_tokens != T, and lookup() raises if it is ever reached while prestage is enabled.

Point 4 is the part worth copying into your own work. A silent fallback here yields a model that serves fluent, confident tokens off stale rows, and you would ship it. Both paths fail loudly instead. The parity mode (DSV41_ENGRAM_PRESTAGE_CHECK=1) ran 190 decode steps including a 3234-token prefill with zero mismatches before graphs went back on.

Trap: the top-level object vLLM hands you is the VL wrapper (vl_model.py); the inner model is model.language_model.model. Our first run failed with covered -1 tokens until the resolver walked language_model.

What this is not

It would be easy to call this a two-tier RAM/NVMe cache. It is not, and the distinction matters if you are reproducing it.

There is no explicit RAM hot tier in this patch set. Rows come off NVMe every step. What the host does have is ~107 GB of page cache holding the hot parts of shards 47/48, because the kernel is good at its job and nothing else wants that memory. That emergent tier is doing real work — the 0.87 ms warm p50 is page cache, not the device — but it is the OS's decision, not ours.

We designed a bounded explicit hot tier and then did not build it, because the profile says the remaining gap to pinned RAM is per-step overhead (hash kernel + one host sync + Python-level gather + dequant), not row latency. Profile that before you buy DIMMs; it is what cancelled ours.

Limitations and known gaps

• 71% of pinned-RAM single-stream throughput. If you have 280 GB of host RAM, use the stock path; this is for people who don't and can't.
• The remaining gap is unprofiled at the kernel level. The Python-level gather is the obvious next target.
• Tested on one host, one pack, one image tag (deepseekv41-flash-0909), thinking off, 2K-context bench. Long-context behaviour beyond a 3234-token prefill is unmeasured.
• The 288-row (k=5 lookback) path measured in the table is not the shape this patch serves; it is included because it bounds the cost if you increase the lookback.
• No upstream runtime has an on-disk Engram mode as of 2026-09-12: vLLM main's engram.py offers cpu_offload (pinned host RAM over UVA) and open PR #56512 adds /dev/shm shared tables — still RAM; SGLang's dsv4.1 branch has GPU / pinned-host / memfd-shared / private-anon backends, all RAM, and fadvise(DONTNEED)s the safetensors after load; llama.cpp PR #28696 is a converter draft with no V4.1 runtime. If any of that lands, prefer it over this.

Credits and licence

The pack, the sm_120 overlay and the DSV41_ENGRAM_DISK code this is derived from are diffbot's (diffbot/DeepSeek-V4.1-Flash-EXL3-2.0bpw-2x-RTX-PRO-6000, MIT). The model is DeepSeek's. The patched files derive from vLLM (Apache-2.0) as shipped in the day-0 image. The prestage hook, the parity check and the fail-loud guards are ours. See NOTICE.md.

There was also a DGX-Spark fork that pre-read Engram rows before the graphed forward — the same idea. Its repo started returning 401 mid-project and we had no local copy, so everything here was re-derived from diffbot's disk code. Mirror your references.

How this was built: four models in a review ring

This work was done inside icarus, our own agent harness, which routes work across cloud and self-hosted models and seats several of them in a shared ring where they see and challenge each other's output. Four models contributed, and all four are credited — including where they were wrong, because that is part of the method:

Seat	Where it runs	Contribution
Fable	cloud, long-context	The runtime survey, the prestage patch set, the parity harness, the benchmark runs.
Astra	cloud	The maintenance-window script (start / serve / smoke / restore) and the ring verdict that deferred the RAM purchase: no hot tier, profile per-step cost first.
Qwen3-Next-80B	self-hosted, llama.cpp	Bounded scripted work. Also produced a fabricated capacity analysis — invented parameter count, invented 420 GB Engram figure, a non-existent arXiv id, a non-existent "internal doc" — which was caught and discarded. Later, asked about code it could not see, it correctly answered "cannot verify, no reference in workspace".
GLM-5.3-Flash	self-hosted, llama.cpp	Independently concurred on deferring the RAM. Also collected "host facts" by running the commands on the wrong machine and reporting them as the target's — real numbers, wrong host, caught. GLM is the model DeepSeek evicted from these two GPUs; it reviewed its own replacement.

Neither failure was caught by a model noticing another model was wrong. Both were caught by a process rule, which is the transferable part of this: small models get scripts to run, never numbers to derive, and you verify which host answered. Every number in this repository traces to a script whose output we kept, or is labelled as reported by someone else.

Published by nkt-labs — nktlabs.ai — as nktlabs/dsv41-flash-engram-disk-prestage.

---

Hardware: 2× RTX PRO 6000 96 GB (sm_120), TRX50, 125 GB DDR5-5600, PCIe 4.0 ×4 NVMe. Software: vLLM day-0 deepseekv41-flash-0909 + sm_120 overlay, EXL3 2.0bpw pack, thinking off. Measured 2026-09-12.