DeepSeek-V4-Flash-0731 — hipfire MQ2-Lloyd

DeepSeek V4 Flash (0731), quantised to MQ2-Lloyd for hipfire. 43 layers, 256 routed experts + 1 shared, top-6 per token, arch_id=9.

file	size	what
deepseek-v4-flash-0731.mq2lloyd	86.2 GB	trunk
deepseek-v4-flash-0731-dspark.mq2lloyd	6.0 GB	DSpark 3-stage draft sidecar
deepseek-v4-flash-0731-adapter-r128.bin	44.6 MB	expert-prediction adapter (experimental, off by default)

The 0731 checkpoint ships the DSpark draft chain in place of the classic nextn MTP head, so it pairs with a -dspark sidecar, not a -mtp one. temperature=1.0 mirrors the base checkpoint's serving guidance.

Running it when the experts do not fit

The trunk needs ~86 GB. Routed experts are 72.2 GiB of that, and can be paged from disk instead of held resident:

export HIPFIRE_DEEPSEEK4_EXPERT_CACHE_GB=auto   # or an explicit GiB budget

auto takes whatever fits after non-expert weights, KV and headroom, clamped against MemAvailable. If everything fits it disables paging rather than paging pointlessly.

Cache budget is the main performance dial

Measured on Radeon 8060S / gfx1151 (Strix Halo), ds4 MQ2 decode:

budget	slots/blob	hit rate	tok/s	ms/token	expert I/O share
4 GiB	14	40.5%	7.02	142	—
8 GiB	28	59.4%	7.06	141.7	41.9%
16 GiB	56	74.4%	—	—	—
32 GiB	112	86.5%	9.09	110.0	26.5%
64 GiB	224	91.0%	—	—	—
resident	256	91.3%	—	—	—

8 GiB → 32 GiB is +28.8% throughput. Hit rate saturates at 91.3% (the rest is compulsory cold-miss), so past ~32 GiB the curve flattens hard — 64 GiB buys only another 4.5pp. Pick 32 GiB if you have it.

Where the time goes at 8 GiB, from ablation (routed MoE off vs on):

component	ms/token	share
expert paging I/O	59.4	42%
routed expert GEMV compute	14.6	10%
attention, shared experts, norms, lm_head	67.7	48%

The routed-MoE path costs 74 ms/token and 80% of that is I/O, not compute — this is an I/O-bound workload wearing a compute-bound costume.

Expert-prediction adapter (experimental — OFF by default)

deepseek-v4-flash-0731-adapter-r128.bin predicts layer L+1's expert selection from layer L's hidden state, so a paged runtime can start fetching a layer early rather than stalling.

It currently makes decode slower. Published for reproduction, not as a win. Every configuration measured came out below 1.0x. Off unless you explicitly set HIPFIRE_DEEPSEEK4_EXPERT_ADAPTER.

How it works

z_{L+1}  ~=  B · (A · h_L)  +  gate_bias_{L+1}       A: [128,4096]  B: [256,128]

Rank, take top-M as prefetch candidates. The frozen native router still makes the real selection — the adapter only decides what to stage, so a wrong prediction costs a wasted fetch, never a wrong token. Output is byte-identical with it on or off (verified, greedy, 9/9 arms).

Fitted by closed-form ridge on 113,433 wikitext-prefill positions, 80/20 split, SVD-truncated to rank 128. 22.3M params, ~22M MACs/token (<1% of per-token compute).

Accuracy

ExpertRecall@M — of the 6 experts actually chosen, how many appear in the top-M:

top_M	recall	covers of 6	wasted	fetched late	total fetches vs baseline
4	59.6%	3.58	0.42	2.42	1.07x
6	75.9%	4.55	1.45	1.45	1.24x
8	83.2%	4.99	3.01	1.01	1.50x
12	89.2%	5.35	6.65	0.65	2.11x

Training-free alternative (running the real gate_{L+1} on h_L): 27.6%. ds4's mHC replaces the residual stream with 4 Sinkhorn-mixed streams, so there is no slowly-evolving residual to read through and a trained map is needed.

Rank matters more than in published work — SpecPrefetch reports ~84% at r=32 on 64-expert DeepSeek-VL2, but ds4 is 256-choose-6 and reaches only 54.9% at r=32:

rank	params/layer	total	recall@6
32	139,264	5.6M	54.9%
64	278,528	11.1M	66.9%
128	557,056	22.3M	75.6%
full	1,048,576	41.9M	78.7%

Why it is not a win

Best result at each cache size, 2-3 reps, interleaved:

cache	best top_M	speedup
4 GiB	3	0.89x
8 GiB	4	0.93x
32 GiB	6	0.86x

No cache size makes it positive, and tightening to 4 GiB made it worse — the "constrained caches are where prefetch pays" hypothesis was tested and refuted.

The cause, measured directly on the same workload:

off:  70,736 accesses,  25,218 misses,   83.1 GiB read
on : 147,200 accesses,  52,900 misses,  174.4 GiB read   (2.10x)

Speculation more than doubles bytes read. Hit rate stays flat at ~64% while bytes double — churn, not caching: staged experts evict entries the real dispatch then needs. No amount of overlap pays for 2.1x the I/O.

Prefetch overhead is a fixed per-layer cost while its benefit scales with how much I/O there is to hide, so it competes with the cache budget rather than composing with it. Raising the budget is strictly the better lever.

Usage

export HIPFIRE_DEEPSEEK4_EXPERT_CACHE_GB=8
export HIPFIRE_DEEPSEEK4_EXPERT_ADAPTER=/path/to/deepseek-v4-flash-0731-adapter-r128.bin
export HIPFIRE_DEEPSEEK4_PREFETCH_TOPM=4     # default k+2=8; 4 measured least-bad

Load errors are fatal, not silent — running without the adapter you asked for would quietly benchmark the wrong configuration.

Where it might still pay (untested)

• Discrete GPUs (R9700 gfx1201, 7900 XTX gfx1100). On unified memory the H2D half is a plain memcpy with nothing to hide; on a discrete card it is a real PCIe DMA with an engine to hide it behind, so the trade may invert.
• Removing the per-dispatch host round-trip. The pager is host-driven, so every dispatch copies routing back to the host — including the ~64% that are pure hits needing no host action.

Format

magic "HFADPT\0\0"  8 B
u32 version=1, u32 n_entries, u32 d_model=4096, u32 n_exp=256, u32 rank=128
per entry: u32 src_layer (predicts src_layer+1), f16 A[rank,d_model], f16 B[n_exp,rank]

Trained and evaluated on wikitext only — cross-domain generalisation unmeasured.