ruisv/bllm-qwen3.5-4b

Qwen3.5-4B — BPU-native build for D-Robotics RDK S100 / S100P

Compiled, quantised (int8 weights / int16 per-linear-calibrated activations), and board-verified build of Qwen/Qwen3.5-4B for D-Robotics' Nash BPU, running entirely through BLLM — an on-device runtime that talks to the BPU directly (hbDNN/hbUCP), with no GPU/CUDA and no cloud round-trip.

This is a compiled BPU binary, not a transformers-loadable checkpoint — closer in kind to a GGUF quantisation than a HF AutoModel. There is no Inference API widget on this page; see "How to run" below.

New here? Start with BLLM (the runtime) and bllm-model-zoo (how this build was made, full acceptance data, what was tried and rejected). This repo is the downloadable artifact; that repo is the method.

Quick facts

Base model	Qwen/Qwen3.5-4B (Apache-2.0)
Architecture	Hybrid: 24 Gated-DeltaNet layers (decoupled 16 key / 32 value heads) + 8 full-attention layers
Quantisation	int8 weight (per-channel) / int16 activation, per-linear calibrated (flat ranges clip this size — see below)
Target hardware	D-Robotics RDK S100 / S100P (Nash nash-m BPU)
Modality	Image + text (vision tower included)
BPU utilization	100% — CPU_inference_time_cost == 0 on the text decoder
Runtime	BLLM (native, no libxlm)

This architecture (linear-attention SSM + head_dim=256 + decoupled GDN key/value head counts) is not covered by D-Robotics' official OE-LLM toolchain (libxlm) — this build exists because BLLM's native runtime compiles the graph directly.

Files in this repo

s100p/
  ctx4k/    # cache_len=4096 — default, recommended context
    model.hbm            # text decoder (prefill+decode dual-graph)
    model_prefill.hbm     # chunked-prefill graph (N=32)
    model.json
    embed_tokens.bin
    tokenizer.json
    visual.hbm            # vision tower, 320px bucket, 100 tokens/image
  ctx512/   # cache_len=512 — faster decode, short-context use
    ...same layout

cache_len is a performance knob as well as a capacity one — a larger KV window costs more DDR traffic per decoded token. Pick ctx512 if your prompts/replies genuinely fit in ~512 tokens; it decodes faster than ctx4k.

How to run

conda install -c https://mirrors.ruis.ai/conda -c conda-forge bllm

import bllm
m = bllm.load("s100p/ctx4k")
print(m.chat("这张图里有什么?", images=["photo.jpg"]))

# text-only, without loading the vision tower into memory
m = bllm.load("s100p/ctx4k", vision=False)
print(m.chat("介绍一下你自己"))

Requires D-Robotics RDK S100/S100P, sw >= 4.0.5, and an enlarged ION carveout (hb_switch_ion.sh balanced or bpu_first for this size + reboot) — see BLLM's docs/LLM_ONBOARD.md.

Measured on hardware (S100P, performance mode, 3 repeated runs)

variant	decode	TTFT (image, coco-cats test photo)	prefill↔decode parity	ppl	image contrast	task anchors
ctx4096	6.14 tok/s	3.29 s	PASS	35.29	PASS	PASS
ctx512	7.54 tok/s	2.93 s	PASS	35.09	PASS	PASS

Decode is nearly 2x the pre-rewrite baseline (3.29 tok/s) — see "Known limitations" below for what changed and why that isn't a free lunch. Full methodology, acceptance protocol (prefill↔decode parity + perplexity + task-coherence anchors + with/without-image contrast — not just cosine similarity, see why in the model-zoo README), and every rejected build are in bllm-model-zoo/models/qwen3.5-4b and .../qwen3.5-vlm.

Known limitations

• This release changed 4B's attention implementation from an expand_win broadcast to GQA batched matmul + mask deduplication (the same rewrite applied to the 0.8B/2B siblings) — this is where the ~2x decode speedup above comes from (most of expand_win's redundant DMA is gone). Unlike the 0.8B/2B siblings, this introduces new int16 quantisation on the attention score/output matmul that the prior expand_win path never had — board output after the rewrite is not byte-identical to the pre-rewrite build (still coherent and correct, e.g. one prior answer mentioned an extra detail the new build's phrasing omits). The batched-matmul math itself was checked algebraically exact against the old math offline (cosine 1.0000001, max abs diff 2.38e-7 on this model's exact head configuration), and this release's full acceptance run (table above) is what actually clears the build for use — the offline math check alone is not a substitute for it.
• Flat (uncalibrated) int16 activation ranges — which fit the 0.8B/2B siblings — clip this size's deep linears into a degenerate loop; this build uses per-linear calibration instead (see the model-zoo recipe for the measured activation ranges and why).
• The all-BPU chunked-prefill path (image ingestion, no CPU fallback) is numerically solvable but impractically slow to compile — this build's prefill graph uses a float [C,C] CPU fallback for one small intermediate instead.

License

The compiled .hbm is a derivative of the upstream Qwen3.5-4B weights and inherits its Apache-2.0 license — check the upstream model card for the current terms. BLLM (the runtime that loads this file) is open source; see its repository for its own license.