granite-4.0-h-350m — LiteRT-LM

ibm-granite/granite-4.0-h-350m converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 — the Mamba2 conv/SSM state buffers are bound through 0.15's generalized state binding (ExecutorMetadata); the 0.14 engine cannot bind them. Companion to litert-community/granite-4.0-h-1b.

Granite 4.0-h is IBM's hybrid architecture: 28 Mamba2 (selective-scan) blocks interleaved with 4 grouped-query attention blocks. The Mamba2 blocks carry per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV.

File	Recipe	Size
granite-4.0-h-350m_fp16.litertlm	fp16 weights (float-casting on linears + embedding; convs/SSM fp32)	723 MB
granite-4.0-h-350m_int8.litertlm	int8 dynamic weight-only (linears + embedding; convs/SSM float)	436 MB

Pick fp16 for reference-faithful quality (greedy outputs match the PyTorch model token-for-token on our probes), int8 for size and speed. On phones we recommend int8: the CPU runtime unpacks fp16 weights to fp32 in RAM (~3.7 GB peak on iPhone vs ~2.1 GB for int8), so fp16 is best treated as the desktop/quality variant.

Correctness

The converted graph is verified against the PyTorch reference at the logits level: a float export matches the HF model exactly at every decode position (8-step teacher-forced comparison, per-position max|logit diff| ≤ 3.9e-4, correlation 1.000000, top-1 identical at all positions) — both the chunked prefill path and the single-step decode path (rolled conv window + SSM state recurrence) reproduce the reference math. Both files pass an 8-question sanity gate at 8/8, matching the HF reference, and a prompt-length sweep (12–200 tokens through the engine's real prefill chunking) generates cleanly — with one caveat for int8 below.

Unlike the HF generate() default, the bundle does not prepend a BOS token — this matches Granite's official chat template, and at 350M scale it matters: a prepended <|end_of_text|> measurably degrades greedy answers on this model.

Known int8 limitation: prompts whose chat-templated length lands exactly on 33–37 tokens can end the reply after a few tokens (a quantization-noise interaction with one prefill chunk shape; the fp16 file is unaffected at all tested lengths). Adding or removing a word avoids it.

Usage

litert-lm run ./granite-4.0-h-350m_fp16.litertlm --prompt "What is the capital of France? Answer in one sentence."

The bundle carries the tokenizer and Granite's full chat template. Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; the KV budget for the attention layers allows up to 4096 tokens. CPU backend — the current GPU delegates reject the scan graph (SLICE rank > 4).

Speed

Device	File	Prefill	Decode	TTFT	Peak RSS
MacBook Pro (M4 Max), CPU	int8	122 tok/s (256)	25.1 tok/s	2.1 s	—
MacBook Pro (M4 Max), CPU	fp16	114 tok/s (256)	22.0 tok/s	2.3 s	—
iPhone 17 Pro, CPU	int8	~10 tok/s (41-tok prompt)	18–21 tok/s	~4 s	~2.1 GB
iPhone 17 Pro, CPU	fp16	~6 tok/s (41-tok prompt)	7.8 tok/s	~10 s	~3.7 GB

(Cold-start single runs. Decode is dominated by the selective-scan running as generic float ops; per-parameter it is slower than a plain transformer of this size.)

Reproduce

Conversion script, litert-torch patch (Mamba2 export-cache layer + decode-state continuation + prefill pad guard), and gate harnesses: https://github.com/john-rocky/hf-to-litertlm (granite_work/).