zotowata/bonsai2-dflash2-fastfork

Bonsai-2-27B DFlash2 fast-fork — 740 tok/s from a 27B dense model on one RTX 5080

This is a build-and-benchmark package, not a new checkpoint: four small patches for the PrismML llama.cpp fork (branch prism, prism-v7 snapshot), a tuned speculative-decoding server profile, a benchmark harness, and the deployment kit that keeps it running 24/7 under WSL2.

Headline, measured live on the deployment (RTX 5080 16GB, WSL2, CUDA graphs on):

Metric	Value
Decode, warm steady state (canonical code-v1, hybrid)	894.9 tok/s engine, 99.9% draft acceptance, 768 tokens in 1.03 s
Decode, warm (canonical reasoning-v1, pure n-gram)	854.5 tok/s, 100% acceptance (750/750) — Verified
Decode, warm (canonical code-v1, pure n-gram)	625.9 tok/s @ 95.7% — Verified
Decode, fresh state (canonical code-v1)	142 tok/s @ 60.6% acceptance
Decode, fresh state (canonical reasoning-v1)	146.4 tok/s @ 61.9% acceptance
DFlash2 only, no n-gram layer (canonical code-v1)	151.2 tok/s @ 60.6% acceptance
TTFT	172 ms warm / 805 ms fresh (code-v1, 201-token prompt)
Prefill	794 tok/s hybrid steady / 1381 tok/s pure n-gram
Peak VRAM	13.7 GB hybrid / 11.3 GB pure n-gram (ctx 32768)
GPU power	165–240 W during generation (peak 268 W)

Leaderboard standing (localmaxxing.com, RTX 5080): #1 and #2 overall (894.9 hybrid / 854.5 pure n-gram — the latter Verified), and five of our seven runs are Verified, including every speed class: warm fast (854.5 / 625.9), fresh-state (146.4 / 142.0), and the DFlash2 isolation (151.2). The fastest verified 20B+ results on this GPU by a wide margin. The next-best 20B+ model on this GPU is gpt-oss-20b (MoE) at 222 tok/s; the fastest sub-2B models reach ~450 tok/s. Fresh-state and warm-state runs are both submitted and labeled — see "Methodology notes" for what each state means.

The three layers of speed

1. The quant. PQ2_0 (group-128, 2.13 bpw ternary) from dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF — uncensored, 6.8 GB of weights/token traffic, ~6.8 GB/s per token at 740 tok/s sits at ~92% of the RTX 5080's measured DRAM ceiling. Decode is bandwidth-bound; the stack is already at the wall.
2. The drafting. DFlash2 self-speculative decoding with a Q4_K draft head (Bonsai-2-27B-DFlash2-Q4_K.gguf, 1.1 GB, included in draft/) plus the fork's ngram-mod layer with a 96-token window. DFlash2 drafts are cheap (same family, tiny head); n-gram drafting is nearly free and goes from ~58% acceptance on a fresh prompt to 90.9% when the workload revisits content — which real agentic/code-edit sessions constantly do.
3. The patches. Round 10: +11% fresh decode and a much snappier server, see below.

The patches (patches/, unified diff vs the pristine prism-v7 tarball)

#	File	Lines	What it does
0001	src/llama-model.cpp	24	Input-layer GPU offload — the token-embedding row lookup moves off the CPU per-token path; single CUDA split, no host round-trip per token.
0002	ggml/src/ggml-backend.cpp	20	Async graph-input staging — CUDA-graph input buffers are staged on the compute stream instead of synchronously on the caller. Removes a per-token sync stall.
0003	ggml/src/ggml-cuda/mmvq.cu	377	Optional PQ2_0 SoA matvec kernel (Round 11) — structure-of-arrays layout with bias fusion, proven bitwise-identical to the AoS kernel. Default off (no aggregate win at this model's shapes); enable with GGML_CUDA_PQ2_SOA=1.
0004	ggml/src/ggml-cuda/ggml-cuda.cu	49	Remaining Round-10 CUDA-side diff (dispatcher plumbing for the above).

Applied on top of the PrismML fork with CUDA graphs enabled (GGML_CUDA_GRAPHS=ON is load-bearing: +70% vanilla decode on this stack).

Reproduce

# base: PrismML llama.cpp fork, branch `prism` (prism-v7 snapshot)
# apply patches/000*.patch with: patch -p1 < patches/0001-....patch

cmake -B build-graphs -DGGML_CUDA=ON -DGGML_CUDA_GRAPHS=ON \
      -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release
cmake --build build-graphs --config Release -j

# serve (see scripts/serve_bonsai.sh for the full flag set)
./build-graphs/bin/llama-server \
  -m Bonsai-2-27B-PQ2_0-CRACK.gguf \
  --model-draft Bonsai-2-27B-DFlash2-Q4_K.gguf \
  --spec-type draft-dflash,ngram-mod \
  --spec-draft-n-max 5 --spec-ngram-mod-n-max 96 \
  -ngl 999 -ngld 999 -fa on -c 32768 -np 1 -t 16 -tb 32 -b 2048 -ub 512 \
  --host 0.0.0.0 --port 8001

Benchmark: scripts/bench_lm.py <probe.json> <n_runs> — a streaming SSE consumer that reports engine tok/s (server-side timings), wall time, TTFT, prefill rate, acceptance, and samples GPU power/VRAM in the background. scripts/bench_prefill2.py measures clean prefill (first long prompt pays graph capture; the second distinct prompt is the real number). Probe: scripts/bonsai_code600.json (600-token code generation, temp 0, thinking off).

Methodology notes (read before comparing numbers):

• "Warm steady state" = the same generation task re-run on a server whose n-gram map has seen the content (runs 3–6 of a repeat series). This is the real steady state of any agentic loop that revisits a file or conversation.
• "Fresh state" = first run after server start, cold n-gram map: 133 tok/s.
• All numbers temp 0, enable_thinking: false, single slot, no other GPU load.
• Engine tok/s is the server's own predicted_per_second; end-to-end HTTP tok/s is ~5–15% lower (streaming + framing overhead).

Deployment kit (deployment/)

Runs 24/7 on WSL2: bonsai.service (systemd, auto-restart), a Windows session-holder scheduled task (bonsai_keepalive_task.xml) that defeats WSL2's ~20 s idle VM shutdown (this WSL build ignores vmIdleTimeout), and start_bonsai.cmd / stop_bonsai.cmd helpers. The service stages serve_bonsai.sh through tr -d '\r' on every start so CRLF can never poison it.

Verification

• 0001+0002+0004: deterministic outputs byte-identical to the prebuilt fork binary on prose; one known greedy near-tie flip on the code probe (float non-associativity in GPU dequant rounding flips a last-bit tie — documented, not a correctness bug).
• 0003: scripts/parity5.sh / parity6.sh prove bitwise parity of the SoA kernel vs the AoS kernel across on/off/default configs under the full production config (draft + graphs).
• nsys per-kernel profiling of the whole matvec stack: ~874 GB/s average = 92% of the measured 949 GB/s DRAM ceiling. The kernel lever is exhausted; remaining per-token cost is ~2.4 ms of non-matvec ops (FWHT, norms, get_rows) and ~0.9 ms inter-graph gap.

Contents

patches/     4 unified diffs vs pristine prism-v7
scripts/     serve profile, bench harness, probes, parity + A/B scripts
deployment/  systemd unit, WSL keepalive task XML, Windows start/stop
results/     raw benchmark JSON + summary
draft/       Bonsai-2-27B-DFlash2-Q4_K.gguf (1.1 GB draft head)

Credits & links

• Base model: dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF (Apache-2.0)
• Draft head lineage: ProCreations/Ternary-Bonsai-2-27B-DFlash2
• Engine: PrismML llama.cpp fork (branch prism, DFlash2 + PQ formats)
• Benchmarks submitted to localmaxxing.com

License: Apache-2.0 (matches the base model and the fork).