Kev-0.8B-RKNN3-RK1828

Kev-0.8B (decision model) converted to RKNN3 and running on the Rockchip RK1828 NPU — with zero accuracy loss versus the original fp32 model.

Kev is not a chat model. It runs a single prefill and scores options with a pointer head; it never generates tokens. You can verify this yourself: ask it 1+1, and you get nonsense. That is the model behaving correctly.

This repository contains the converted artifacts plus the minimal host-side code needed to actually use them, because a plain rkllm3-server deployment cannot run Kev (see Why this needs host code).

Base model	Qwen/Qwen3.5-0.8B-Base @ dc7cdfe
Adapter	jaredpalmer/kev-0.8b (LoRA r16, merged into base)
Target hardware	RK1828 NPU (RK1820/RK1828 AI SDK), host = Windows PC / RK3588 / RK3576
Toolchain	RKNN3 v1.1.0 (runtime/rknn3_110), target_platform='rk1820'
Quantization	w4a16, quantized_algorithm='normal', quantized_method='group32' (weight-only, no calibration set)
NPU footprint	802 MB (as reported by rknn-smi)
Latency	0.9 s per decision end-to-end, ~75 ms per prefill (100 tokens)

---

Files

File	Size	What it is
rknn/Kev-0.8B.rknn	20.6 MiB	RKNN3 graph. Output is [1, 1, 248320] fp16 logits
rknn/Kev-0.8B.weight	498 MiB	Quantized backbone weights
rknn/Kev-0.8B.embed.bin	485 MiB	Token embedding table, 248320 × 1024 fp16 (also the dequantization matrix W)
rknn/Kev-0.8B.tokenizer.gguf	10.4 MiB	Tokenizer. Identical to Qwen3.5-0.8B — LoRA does not touch the vocab
rknn/start.cmd	1.4 KiB	rkllm3-server launch line (for the "text-only" sanity check)
host/kev_daemon.cpp	—	Device-side runner: loads the model once, then takes TOK/R/A commands and dumps logits per request
host/kev_serve.py	—	HTTP service exposing POST /v1/systemone
host/serve_assets/*.npy	6.1 MiB	Ginv.npy (1024×1024), head_q.npy, head_k.npy, temperature.npy — precomputed, so you do not have to invert a 1024×1024 normal-equation matrix at startup
host/e2e_check.py	—	End-to-end accuracy check against the decision-v2 labels
docs/DEPLOY.md	—	Step-by-step deployment, including the traps

Total: ~1.06 GB.

Checksums

776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0  Kev-0.8B.rknn
0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce  Kev-0.8B.weight
2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0  Kev-0.8B.embed.bin
f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f  Kev-0.8B.tokenizer.gguf

sha256sum -c <<'EOF'
776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0  Kev-0.8B.rknn
0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce  Kev-0.8B.weight
2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0  Kev-0.8B.embed.bin
f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f  Kev-0.8B.tokenizer.gguf
EOF

---

Why this needs host code

Kev's scoring path is:

h_decide = backbone(state + question + "<decide>")     # hidden state, 1024-d, *before* lm_head
h_opt_k  = backbone(state + question + option_k + "</opt>")
z_k      = (k(h_opt_k) · q(h_decide)) / sqrt(256)
answer   = softmax(z / T)                              # T = 2.3510958125672174

The runtime exposes vocab logits, not hidden states. The usual conclusion is "blocked, needs retraining". It is not blocked, because logits = h @ Wᵀ is an overdetermined system:

W ∈ R^(248320 × 1024),  unknown h ∈ R^1024,  248320 equations
h = (WᵀW)⁻¹ Wᵀ · logits

W is the (shared) embedding matrix, so it is already on disk. The per-coordinate quantization noise gets averaged over ~248k equations, i.e. suppressed by roughly √248320 ≈ 500×. Measured cosine similarity between device-recovered h and the fp32 reference is 0.96–0.99, with top-1 agreeing on every sampled position.

So the original pointer head runs unmodified on the host. No retraining, no downgrade to an embedding model.

Ginv.npy is the precomputed (WᵀW)⁻¹Wᵀ, so recovery is a single 1024×1024 @ 1024 matmul per position.

Two consequences worth knowing before you deploy:

• The graph only keeps the last position. There is no num_logits_to_keep in the public API and clear_kvcache has no truncate. To read the </opt> positions you must re-run a truncated sequence. A request with K options therefore costs K+1 prefills — that is a hard floor, not an implementation choice.
• Per-call fixed overhead is ~50 ms, independent of length. Latency scales with number of calls, not token count. Batching long states into one prefix helps a lot; splitting into many short calls does not.

---

Usage

1. Sanity check (text only)

cd rknn
start.cmd

Then POST /v1/chat/completions. Output will be garbage — this only proves the weights loaded and the NPU is running. Do not use this to judge whether the conversion worked.

2. Real usage: decision API

Build host/kev_daemon.cpp against the RKNN3 SDK session demo, run it, then:

python host/kev_serve.py          # listens on :17890

curl -X POST http://127.0.0.1:17890/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"ticket": "Checkout returns 500 for EU customers", "age_hours": 30},
    "questions": {
      "team":   {"type": "choice", "instructions": "Which team should own this?",
                 "criteria": {"billing": "payment, invoices, refunds",
                              "platform": "infra, deploys, runtime",
                              "growth": "onboarding, activation"}},
      "urgent": {"type": "noul", "instructions": "Is this urgent?"},
      "sev":    {"type": "score", "instructions": "Rate the severity",
                 "criteria": ["cosmetic", "degraded", "broken", "outage"]}
    }
  }'

{"answers": {
  "team":   {"type": "choice", "answer": "billing", "confidence": 0.7638, "probabilities": {...}},
  "urgent": {"type": "noul", "p_true": 0.912},
  "sev":    {"type": "score", "score": 2.31, "legend": [...], "score_confidence": 0.54}
}}

Response schema follows the upstream kev/api.py::to_answers conventions:

• noul returns p_true only (the upstream API does not binarize it)
• choice returns confidence = (p_max − 1/K) / (1 − 1/K), so K=2 with p=1.0 gives confidence=1.0
• score returns the probability-weighted expected level as score, plus per-level probabilities

Full commands and the build line: docs/DEPLOY.md.

---

Performance

Measured on RK1828 (5051.8 MB NPU), Windows host, single model loaded.

Metric	Value
Accuracy, decision-v2 dev 60 (fp32 reference, host)	49/60 = 0.817
Accuracy, decision-v2 dev 60 (device, w4a16 + recovered h)	49/60 = 0.817
Per-question argmax agreement, device vs fp32	58/60 = 96.7%
End-to-end HTTP, 26 mixed-type questions	23/26 = 0.885
Prefill throughput (long sequences)	1200–1500 tok/s
Single prefill, ~100 tokens	~75 ms
Per-call fixed overhead	~50 ms
One decision (K=5)	~0.5 s
End-to-end HTTP per question	~0.9 s
Model load	~30 s, once
NPU memory	802 MB

The two disagreeing questions (q9, q10) cancel out — one is right on device and wrong on fp32, the other the reverse. That is the signature of the same function being computed, with numerical noise flipping borderline cases.

Coexistence, measured with rknn-smi on the same device:

Configuration	NPU used	Delta
baseline (proxy only)	34 MB	—
Kev-0.8B alone	802 MB	—
Kev-0.8B + Qwen3-ASR-0.6B	2848 MB	+2047 MB
Kev-0.8B + Qwen3-ASR-1.7B	2953 MB	+2152 MB

Kev can share the NPU with an ASR model. Both fit; only ~105 MB separates the two ASR sizes. Measure coexisting configurations, not standalone ones — a standalone run reports 779 MB for the 0.6B ASR, which will make you over-budget your plan.

---

Limitations

• Not a chat model. Text generation output is meaningless by design. Any benchmark that prompts it for answers is invalid.
• Numeric noise flips borderline decisions. 96.7% per-question agreement means ~3% of hard cases can differ from the fp32 reference. If you need exact reproducibility, run the fp32 model.
• K+1 prefill calls per request cannot be reduced with this runtime (see above). Very large option lists get slow linearly.
• Single-request serialization. The service holds a global lock around one decision, because K+1 prefills share one KV cache; interleaving concurrent requests corrupts it. Throughput is therefore ~1 request at a time. Concurrency is safe (tested), but it queues.
• Host must be Windows / RK3588 / RK3576. The RK1828 is a co-processor; an x86 Linux box can convert and simulate but cannot drive the module.
• Quantization format is fixed to w4a16 + normal + group32 by this toolchain. w8a16 and fp16 builds are rejected (request_type=float16 / Failed to create initializer tensor), so you cannot "build a higher-precision version" to isolate quantization error.

---

Provenance

• Base: Qwen/Qwen3.5-0.8B-Base, revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68
• Adapter: jaredpalmer/kev-0.8b, LoRA r16 / α32, task type FEATURE_EXTRACTION
• Merged via merge_and_unload(), then exported to ONNX with the official rknn3-model-zoo/examples/Qwen3_5 pipeline
• h, the pointer head, and the option/decide token ids are all taken from the upstream Kev implementation unchanged

Special token ids used (from upstream kev/model.py::SPECIAL):

<state> 248060   <question> 248061   <opt> 248049   </opt> 248050   <decide> 248062

Artifact sizes match the officially published Qwen3.5-0.8B RKNN3 build file-for-file (20.6 / 498 / 485 / 10.4 MiB), which is the practical proof that the conversion is structurally correct.

License: Apache-2.0, inherited from the base model.

---

中文摘要

Kev-0.8B 是决策模型，不是对话模型：它只跑一次 prefill，用指针头给候选项打分，不生成 token。 用"问它 1+1"来验收是错的——那样一定会得到无意义的回答。

RKNN3 运行时只暴露 vocab logits，而 Kev 需要 hidden state。但 logits = h @ Wᵀ 是超定方程 （248320 个方程解 1024 个未知数），h = (WᵀW)⁻¹Wᵀ·logits 可以精确解回来， 量化噪声被约 500 倍平均掉。因此原版指针头可以原样跑在 host 侧，无需重训、无需改结构。

项	结果
真机准确率（60 题）	0.817，与 fp32 完全一致
逐题一致	58/60
端到端 HTTP（26 题三题型）	0.885，约 0.9 s/题
NPU 占用	802 MB

本仓库包含转换产物 + 可复现的 host 侧代码（daemon、HTTP 服务、反解矩阵、头权重）， 拿到即可跑通。部署步骤见 docs/DEPLOY.md。