granite-4.0-h-1b — LiteRT-LM

ibm-granite/granite-4.0-h-1b converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Mamba2-hybrid LLM served by the released LiteRT-LM runtime — it depends on 0.15's generalized state binding (ExecutorMetadata section); the 0.14 engine cannot bind mamba conv/SSM state buffers.

Granite 4.0-h is IBM's hybrid architecture: Mamba2 (selective-scan) blocks interleaved with a few grouped-query attention blocks (this 1B/1.5B-class model has 36 mamba + 4 attention layers). The mamba blocks carry constant-size per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV (4096-token budget here).

File	Recipe	Size
granite-4.0-h-1b_int8.litertlm	int8 dynamic on linears + embedding (convs and the selective scan stay float)	1.63 GB

Correctness

The converted graph is verified against the PyTorch reference at the logits level: a float export matches the HF model exactly at every decode position (teacher-forced 8-step comparison: per-position max|logit diff| ≤ 1.3e-4, correlation 1.000000, top-1 identical), covering both the chunked prefill path (with cross-chunk state continuation) and the single-step decode path (rolled conv window + SSM recurrence). The published int8 file scores 8/8 on an 8-question sanity gate — identical to the PyTorch reference (8/8) — and passes a prompt-length robustness sweep (first-token check at every prompt length 12–60 against the runtime's real prefill chunk plans, all clean).

Usage

litert-lm run ./granite-4.0-h-1b_int8.litertlm --prompt "What is the capital of France? Answer in one word."

The bundle carries the tokenizer and Granite's full chat template (tool-calling and RAG-documents sections included). Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.

Speed

litert-lm benchmark, CPU backend, Mac M4 Max, max-num-tokens 1024:

Variant	Backend	Prefill (256)	Prefill (1024)	Decode	TTFT
int8	CPU	67 tok/s	74 tok/s	10.8 tok/s	3.9 s

Honest note: this is slower per-parameter than pure-attention models today. The selective-scan lowers to generic ops (CUMSUM/EXP/BATCH_MATMUL) that run in float on CPU, and current GPU delegates reject the scan's high-rank slices — so this release is CPU-only and the scan is unquantized. Treat it as a correctness-first, first-of-its-architecture release; speed has clear headroom (scan kernels, GPU support) on the runtime side.

iPhone / Android device numbers: coming — the table will be updated with on-device measurements.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm granite_work/):

• Export cache for Mamba2 layers: conv [B, conv_dim, K] + SSM recurrent [B, heads, head_dim, state] cache layers registered for Granite's layer types, so torch.export traces the model's own state contract.
• State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/SSM state consumed, so multi-chunk prefill composes), and the decode graph traces the single-step branch (conv window rolled by one). Without this the decode graph has no state continuity.
• Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the SSM (dt forced to ~0) and the stored conv window is gathered at the last valid column via an in-graph one-hot matmul. Without the guard, generation corrupts at chunk-plan-dependent prompt lengths.
• Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an ExecutorMetadata section listing each state tensor; it is appended at package time.
• Quantization: post-hoc dynamic int8 over linears + embedding only. Export-time conv-int8 measurably costs quality on this family (as with LFM2.5's convs), so the convs and scan stay float.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle. This repository is a community conversion and is not affiliated with IBM.