zst50/sdgen2-npu-gguf-test

• testing gguf hexagon on 12gb xiaomi 13

• gemma 12B will not inference with npu:

• Your chip has 2 HTP cores (HTP0/HTP1) — so "2 sessions" is the physical max on 8 Gen 2. "3 sessions" only exists on 8 Elite Gen 5 (v81). GenieX's docs only expose multi-HTP on v75+ chips.

• gemma 12B crashes, lags the ui on gpu and cpu.

• copies gguf files to app storage for tests, gemma-4-e4B works on npu.

The GGUF Q4_0 path on the Hexagon NPU is effectively W4A16 compute:

    Weights: Q4_0 — 4-bit, one FP16 scale per 32-element block, repacked on-chip into the HTP tiled format (32×32 tiles). Dequantized on the fly by the HTP (W_DEQUANT PMU events).
    Activations: FP16 — the HMX (Hexagon Matrix eXtension) engine is a 16-bit FP matrix unit; inputs/activations flow as FP16.
    Accumulation: FP32 — dot-products accumulate in 32-bit inside the matmul, then cast for the output tensor.
    KV cache: Q8_0 (int8) — auto-enabled on the npu/HTP0 path (along with flash-attention).
    Norms/scales: F32 — the small F32 tensors (RMSNorm γ, rope, layer scales) run as FP32.

So: Q4_0 weights × FP16 activations → FP32 accumulate, KV in Q8_0. That's the same w4a16 profile the QAIRT qairt bundles use — the difference is the GGUF path dequantizes Q4_0 blocks in hardware instead of using precompiled int4 context binaries.

doesnt seem to be a difference between hybrid and npu. why?

Because on your 8 Gen 2 they collapse to the same thing:

    hybrid = llama.cpp's per-tensor scheduler across all registered backend devices.
    npu = pin everything to HTP0, unsupported ops fall back to CPU.

The scheduler's advantage only appears when multiple HTP devices exist (HTP0+HTP1…). But we found the source guard earlier: on Android with arch < 75 (v73), opt_ndev is forced to 1 — GenieX registers exactly one HTP session on your chip. So:

    hybrid → puts HTP-supported ops on the only HTP device (HTP0), CPU for the rest.
    npu → same thing, via pinning.

Identical placement → identical speed. There is one hidden difference: pinning HTP0 also enables flash-attn + Q8_0 KV cache (hybrid doesn't get that bonus) — but on your short 8-token test that's invisible. You'd only see it on long-prompt/long-context runs.

On an 8 Elite (v75+), GenieX registers 2–3 HTP sessions, and there hybrid genuinely beats npu (that's the ~30% bump Qualcomm documents). On v73, don't bother switching — they're the same.