Qwen3.5-2B — LiteRT-LM

Qwen/Qwen3.5-2B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0). Same conversion rail as our Qwen3.5-0.8B and Qwen3.5-4B — the GPU-delegable rank-≤4 gated-delta kernel.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 2B model has 18 linear-attention + 6 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 6 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal; this package is the text decoder only (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

File	Recipe	Size
Qwen3.5-2B_int8.litertlm	int8 dynamic on linears + embedding (convs and the delta rule stay float), fp32 activations declared	1.97 GB

Correctness

All gates below were run on this exact file (or its float parent, same graph rail):

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): on the composite 8-question probe (all 8 questions in one 138-token prompt), the GPU output matches the HF fp32 reference word-for-word through answer 7 (including reproducing the model's own arithmetic slip on question 1 — faithful conversion includes the model's mistakes). At that point fp32 ends its turn without attempting question 8; the int8 path diverges by that one end-of-turn token and adds the correct 8th answer before stopping.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.

Usage

litert-lm run ./Qwen3.5-2B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-2B_int8.litertlm --backend gpu --cache no --prompt "..."

Prefill signatures: the full 1–1024 ladder (1024, 512, 256, 128, 64, 32, 16, 8, 4, 2, 1). Every exported signature is charged engine memory even if never called; unlike the 4B (which needs a reduced ladder to stay under a 12 GB iPhone's jetsam ceiling), the 2B's full ladder peaks at ~5.3 GB on iPhone 17 Pro GPU.

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty <think>\n\n</think> block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	1486 tok/s	114.3 tok/s	0.18 s
CPU	592 tok/s	37.6 tok/s	0.46 s

On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	237.7 tok/s	24.3 tok/s	0.73 s	5.33 GB
iPhone 17 Pro	CPU	206.5 tok/s	16.2 tok/s	0.77 s	1.52 GB

(iPhone GPU: model load ~60 s on first launch while Metal programs compile.)

Honest notes:

• GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from.
• Pixel 8a (8 GB-class Android): GPU does not fit. The OpenCL delegate accepts the whole graph (zero rejections, full delegation on every compiled signature), but engine creation ran the phone out of memory before finishing — the GPU path's fp32-expanded weight buffers are several times the 1.97 GB file. A CPU-backend measurement was also blocked on our test phone, by storage rather than RAM: the runtime's XNNPACK weight cache needs roughly another file-size of free disk, which a nearly-full phone doesn't have. Treat this release as Apple-hardware-first; on Android, prefer 12 GB+ devices for GPU.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality: individual questions gate 8/8 on every backend and both runtime versions. On the harder composite probe (8 questions in one prompt), GPU reproduces the fp32 reference; the CPU int8 path degrades (answers 3 of the 8, deterministically, identical on Mac and iPhone). For complex multi-part prompts, prefer the GPU backend or ask for a float variant.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (shared with the 0.8B/4B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• Unlike the 4B, the 2B's linear-attention heads are ungrouped (16 key / 16 value), so the 4B's head-interleave rewrite never traces, and the full prefill signature ladder fits on-phone.