granite-4.1-3b — LiteRT-LM

ibm-granite/granite-4.1-3b converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.16.

Granite 4.1-3B is IBM's 3.4B-parameter dense instruct model (40 layers, hidden 2560, GQA 40:8, tied embeddings, 100k vocab), long-context and tool-calling capable, released April 2026.

File	Recipe	Size
granite-4.1-3b_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding	2.19 GB
granite-4.1-3b_int8.litertlm	int8 dynamic on linears + embedding	3.83 GB

The int4 file is the one to use on a phone: on Apple hardware it is both smaller and faster than int8 (see below), and int8's memory footprint does not fit a handset GPU.

Correctness

The int4 file scores 8/8 on an 8-question sanity gate on both CPU and GPU (Mac, litert-lm 0.16.0), 8/8 on iPhone 17 Pro on Metal (and 7/8 on its CPU — the one miss is the rhyme line in a single combined prompt, which the model answers correctly when asked on its own), and generates correctly on a Pixel 8a with every node on the OpenCL delegate — 1784/1784 in the 1024-token prefill graph and 1614/1614 in decode, zero rejected ops. The int8 file scores 8/8 on all the same Mac lanes and also runs fully delegated on the Pixel.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for every row:

Configuration	GSM8K
PyTorch bf16 (reference, MPS)	88.0%
LiteRT int8	87.0%
LiteRT int4 (this file)	84.0%

int4 costs 4 points on this model — an ordinary blockwise-int4 result, and worth the 1.7× size reduction for a phone.

Usage

litert-lm run ./granite-4.1-3b_int4.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./granite-4.1-3b_int4.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and Granite's role markers (<|start_of_role|>…<|end_of_role|>), with a 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1). Six rather than eleven is deliberate: every exported signature is charged engine memory whether or not it is called, and an eleven-signature build of these same weights was killed by iOS during Metal initialisation unless the app capped the context at 1024. This file creates its engine at the full 4096 on an iPhone 17 Pro.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --cache no, quiet machine, serialized:

File	Backend	Prefill (256)	Decode	TTFT	Init
int4	GPU (Metal)	1241 tok/s	86.3 tok/s	0.23 s	5.96 s
int4	CPU	104 tok/s	22.0 tok/s	3.01 s	12.5 s
int8	GPU (Metal)	1221 tok/s	71.8 tok/s	0.24 s	5.19 s
int8	CPU	257 tok/s	20.5 tok/s	3.06 s	43.2 s

Pixel 8a (Tensor G3, Mali-G715, 7.75 GB RAM; litert_lm_main built from the v0.16.0 tag, single runs):

File	Backend	Prefill	Decode	TTFT	Init
int4	GPU (OpenCL)	20.0 tok/s	7.07 tok/s	0.99 s	32.5 s
int4	CPU	5.40 tok/s	6.98 tok/s	3.29 s	—
int8	GPU (OpenCL)	15.5 tok/s	3.86 tok/s	1.36 s	65.2 s

Honest notes:

• On this low-end phone the GPU wins prefill (3.7×) and TTFT (3.3×), not decode (7.07 vs 6.98 tok/s). Decode is memory-bandwidth-bound and a phone's CPU and GPU share the same LPDDR. On Apple hardware the GPU wins across the board.
• int4 beats int8 on the GPU on both platforms — 1.2× decode on Metal, 1.8× on Mali — while int8 wins CPU prefill (2.5× on Mac), where XNNPACK's large matmuls outrun the blockwise dequant.
• A 3.4B model on a 2020s mid-range phone is not a fast experience in absolute terms; the numbers above are what the hardware gives.

Conversion notes

Converted with litert-torch 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0, from a pristine released stack — no patched checkout (reproduction script: hf-to-litertlm granite41_work/).

• No start token in the metadata. Granite's tokenizer sets add_bos_token: False, and its BOS is its EOS (<|end_of_text|>). The converter writes a start_token from tokenizer.bos_token regardless, and the runtime prepends it on the first turn — which this model reads as a document that already ended, answering by echoing the question back. Removing the field is the whole fix, and the difference is 5/8 vs 8/8 on the sanity gate. The same behaviour reproduces on the bf16 PyTorch model when the token is prepended by hand, so it is a prompt-construction issue rather than anything about the conversion or the quantization.
• Prefill ladder trimmed to six signatures. Every exported signature is charged engine memory whether or not it is called. With the full eleven, engine creation on iPhone 17 Pro was killed by the OS during Metal initialisation; six (1024, 256, 64, 16, 4, 1) leaves comfortable headroom and costs only padded chunks at intermediate prompt lengths.
• Embedder externalised so the tied 100352×2560 vocab table sits in its own section, clear of the ~2 GiB single-section mmap ceiling on iOS.
• Quantization: int4 is blockwise-32 with OCTAV clipping on the linears and int8 on the embedding; int8 is dynamic per-channel on linears and embedding.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle. This repository is a community conversion and is not affiliated with IBM.