granite-4.0-micro — LiteRT-LM

ibm-granite/granite-4.0-micro converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime. Tested on litert-lm 0.18.0 on a Mac; the phone rows ran the prebuilt Android binary named below. On litert-lm 0.17.1 both files answered a plain run with "The capital of France is Paris." (one CPU process each).

IBM describes Granite-4.0-Micro as a 3B-parameter long-context instruct model finetuned from Granite-4.0-Micro-Base, with improved instruction following and tool calling, for assistants and tool-using agents. It is a dense decoder-only transformer (GQA, RoPE, SwiGLU, RMSNorm): 3,402,836,480 parameters, 40 layers, hidden size 2560, 40 attention heads with 8 KV heads, a 100,352-token vocabulary with tied embeddings, a 128K context (4096 in these files); IBM lists twelve languages, from English to Chinese. The checkpoint is published under the hybrid-MoE class with 40 attention layers and no experts; it was rewritten to the plain Granite class before export, weights unchanged (Conversion notes).

File	Recipe	Size	sha256
granite-4.0-micro_int8.litertlm	int8 dynamic on linears, float activations; the embedding table in its own section	3.77 GB	15561bb4f99dcfcb4b689306431bc2be3acc4ece753051d50058276b286c2354
granite-4.0-micro_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding; the embedding table in its own section	2.20 GB	0696b8e8f5aaf3b50a94baa5a844df3928138e15bceb711a25dacb35c213fb48

On GSM8K (n=100) bf16 scored 86, the int8 file 83 on the CPU and 90 on the GPU, the int4 file 84 on the GPU (table in Accuracy); the int4 file is 41.7 % smaller. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int4 GPU	3.64	21.25	41.50
int8 GPU	4.38	27.76	no row (Performance)
int8 CPU	17.81	53.09	186.29
int4 CPU	20.65	62.08	178.95

So int4 is the faster file on every GPU; on the CPU, int8 is faster on the Mac and the Galaxy S26 and int4 on the Pixel 8a. Not measured on iOS: the int8 file's main weights section is 3,501,542,592 bytes (3.26 GiB), over 2 GiB; the embedding table sits in a separate 265,343,328-byte section, as in the granite-4.1-3b conversion, whose card explains the iOS reason.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, --temperature 0, one litert-lm run process each): int8 8/8 on CPU and GPU, int4 8/8 on CPU and GPU, bf16 (transformers) 8/8.

All 7 decoder signatures of both files were fully delegated on the Mac GPU (each 4-op embedding signature runs one op on the CPU), and every phone GPU leg below ran fully delegated. On both phones, both files answered "What is the capital of France? Answer in one sentence." on GPU and CPU: "The capital of France is Paris." A prompt-length check (39 lengths from 40 to 100 tokens, a fresh engine each, both files, Mac CPU and GPU) returned the expected word every time, and a three-turn conversation on both files and both backends carried the first turn's word into the second and rewrote the second as a question in the third.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, the bundle's own template, greedy, at most 1024 output tokens, a fresh process per question (litert-lm 0.18.0 Python API); bf16 in transformers on MPS with the same prompts. IFEval-lite was not run for this model. Scores are comparable only within this table, not with IBM's 85.45 (8-shot, a different protocol).

Configuration	GSM8K (n=100)
bf16 reference (transformers, MPS)	86
int8, litert-lm CPU	83
int8, litert-lm Mac GPU	90
int4, litert-lm Mac GPU	84

Against bf16, 11 questions differ for int8 on the CPU (7 right only in bf16, 4 only in int8), 8 for int8 on the GPU (2 / 6) and 10 for int4 on the GPU (6 / 4). The files run the GPU with the runtime's default fp16 activations: the int4 file scored the same (84) with fp32 activations declared, and the int8 file's GPU score is not below its CPU score.

Usage

litert-lm run ./granite-4.0-micro_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./granite-4.0-micro_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./granite-4.0-micro_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./granite-4.0-micro_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm benchmark ./granite-4.0-micro_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu

All four run commands answered with the two dates ("The dates extracted from the text are May 3 and May 17."; the int4 file on the CPU: "The dates mentioned in the text are May 3 and May 17."), with no chat marker and no thought block. Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model (3.42 GB for int8, 1.92 GB for int4). The bundle sets no sampler; run passes one only when you give --top-k, --top-p, --temperature or --seed. For Android, see android-gpu.md.

The chat template is IBM's chat_template.jinja byte for byte, with its default system message ("You are a helpful assistant. Please ensure responses are professional, accurate, and safe."), replaced when you pass one; no start token (the prompt begins with <|start_of_role|>), the only stop token <|end_of_text|> (id 100257), no thinking declared. The runtime renders nine of ten conversation shapes exactly as transformers does. The exception: a tool result passed as a Python-API content part renders empty and the model invented a weather report; passed as the tool message's text it was used ("22 degrees Celsius and sunny" in both files).

Tool calling: with a get_weather tool both files wrote <tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call> as text, which the app has to parse (the bundle's generic model type has no tool-call parser), and IBM's documents turn renders for documents passed as context. Engine reuse (int8 file, Python API, CPU, greedy): a second conversation on an Engine that had served another gave the same replies as on a fresh Engine, including with the shared-prefix condition of LiteRT-LM #3165 (3 of 3 runs); this model keeps only a KV cache, no recurrent state.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two passes per file, the median of the two; GPU is WebGPU on Metal. Init is the first iteration's engine init.

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	1424.81	69.83	0.733	4.28
int4 GPU	1454.79	87.28	0.715	4.60
int8 CPU	186.19	20.79	5.550	23.97
int4 CPU	103.22	23.87	9.964	6.85

Phones, Android 16: Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno 840, MemTotal 10.9 GiB) and Pixel 8a (Tensor G3, Mali-G715, MemTotal 7.4 GiB), the same workload through the prebuilt litert_lm_advanced_main (sha256 adac974b…, 40,382,152 bytes) published 2026-09-18 in LiteRT-LM's public binary bucket (binaries/latest/android_arm64/litert_lm/), with --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true. Rates are the mean of iterations 2 and 3 (GPU: LITERT_CL); in 19 of the 20 phone benchmark legs the third iteration was slower than the second. Init cold = the first process after the compiled-program caches were removed, warm = the benchmark process; peak = the largest of the benchmark process's Peak system ram usage lines.

Galaxy S26	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	288.9	10.57	3.72	8.03 / 6.21	976.0
int4 GPU	290.0	14.45	3.67	10.98 / 8.80	1471.0
int8 CPU	68.8	6.70	15.09	4.12 / 0.26	5664.7
int4 CPU	34.6	7.88	29.82	4.90 / 3.85	4373.5

Each GPU row is one of three runs in different holds; over the three the workload took 27.75–29.17 s (int8) and 21.25–22.17 s (int4). The clocks lowered during every run: the GPU clock ceiling fell from 1300 MHz to 422–500 MHz inside the benchmarks, and the clock sampler read a lowered clock in 31–40 of its 34–43 samples (GPU runs) and in 68 of 69 (int8) and 81 of 83 (int4) samples (CPU runs).

Pixel 8a	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	—	—	—	—	—
int4 GPU	98.2	8.24	10.55	33.86 / 16.75	2241.6
int8 CPU	16.2	2.08	63.89	8.97 / 2.56	5024.8
int4 CPU	11.7	2.80	88.17	12.39 / 24.84	3704.8

The int8 file has no Pixel 8a GPU row: the harness stops a process when the phone's MemAvailable falls below 400,000 kB, and it did so in every process of the one try (307,820 kB in the first process, 381,800 kB in the benchmark, 106,656 kB in the prompt process). In the CPU runs the CPU clock was lowered in every sample (180 of 180 int8, 186 of 186 int4); the sampler reads only Adreno GPU clocks, so no Pixel 8a GPU clock was recorded. Not run on the NPU.

Conversion notes

• Source and stack: ibm-granite/granite-4.0-micro at revision 56111ae1 (every file checked against the Hub's record); litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1, torch 2.13.0; packed by the litert-lm 0.18.0 CLI. granite_work/convert_granite40_micro.sh in hf-to-litertlm rebuilt both files, all five sections byte-identical (only the packer's uuid and timestamp differ).
• Dense rewrite: the checkpoint's class is GraniteMoeHybrid with 40 attention layers and 0 experts, which transformers 5.14.1 cannot run through generate(). The recipe copies it into the GraniteForCausalLM layout (each layer's shared_mlp.input_linear rows 0–8191 become gate_proj, rows 8192–16383 up_proj, output_linear becomes down_proj; the other 242 tensors unchanged) and proves it on three prompts of 37, 77 and 839 tokens in fp32: max |logit delta| 1.9e-4, 3.3e-4 and 9.1e-4, the argmax equal at every position, 50 greedy tokens identical, delta 0.0 in float64.
• Direct export: exporting the original class also worked (a fused gate|up matmul plus two slices, 3,771,027,808 bytes) and ran within 3 % of the rewrite on the Mac; the rewrite shipped.
• Order: export of the rewritten checkpoint (6 prefill signatures, 1024 down to 1, plus decode, over a 4096-position KV cache; the embedding table as its own section; no start token; IBM's template embedded) with the int8 recipe or the int4 recipe → int4 block-scale scan (281 blockwise tensors, none zero or subnormal) → metadata edit (stop token id 100257 only, max_num_tokens 4096, no thinking or function-calling flag; the exporter's 14 string stops dropped) with every other section proven unchanged.

License and changes

Distributed under the Apache License 2.0 inherited from the base model (the model repo carries the license in its README metadata and no separate LICENSE file). Changes from the original work: rewritten from the GraniteMoeHybrid class to the GraniteForCausalLM class with the weights unchanged, exported to the LiteRT-LM format and quantized to int8 and int4 as described above; chat template IBM's own, stop token reduced to <|end_of_text|>, max_num_tokens 4096. This conversion was made and is published independently of IBM.