LFM2.5-230M — LiteRT-LM

LiquidAI/LFM2.5-230M converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Tested on litert-lm 0.16.x; the bundles carry the ExecutorMetadata section that litert-lm ≥ 0.15 requires for this hybrid architecture.

LFM2.5-230M is Liquid AI's smallest instruction-tuned model (230M parameters, 8 gated short-conv blocks + 6 GQA attention blocks, tied embeddings, 65k vocab). Liquid positions it for data extraction and lightweight on-device agentic pipelines, not for math or heavy reasoning — our measurements below agree.

File	Recipe	Size
LFM2.5-230M_int8.litertlm	int8 dynamic on linears + embedding, convs float	266 MB
LFM2.5-230M_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding	177 MB

int8 is the recommended file. It is closer to bf16 on instruction following, and at this model size it is also the faster file on Android GPU (2.8× faster decode than int4 on the Galaxy S26) and equal on desktop. int4 is 89 MB smaller and decodes 13% faster on iPhone Metal — pick it for size or iPhone-only deployments.

Correctness

8-question sanity gate (Apple M4 Max, litert-lm 0.16.0 lineage): int8 7/8 on CPU and 7/8 on GPU; int4 6/8 on CPU and 8/8 on GPU, no degeneration on any leg. The bf16 PyTorch reference scores 8/8 on the same questions. int8's one miss is a rhyme-completion line; int4's CPU leg misses that plus one translation question — a real but small quantization cost.

Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno): both files generate correctly on GPU and CPU, with full OpenCL delegation — every prefill signature and the decode graph replaced by LITERT_CL with zero rejected ops (492/492 nodes on each prefill signature).

iPhone 17 Pro: both files pass the same generation check on Metal and CPU (clean stop, thermal nominal throughout).

Accuracy

The model's headline skill is instruction following, so the quantization was chosen on it. We ran the IFEval prompts through a re-implementation of the mechanically checkable instruction types (n=120, prompt-level strict, greedy). These numbers are comparable only within this table — the checkers are simplified, so do not compare them to published IFEval scores:

Configuration	IFEval-style strict	GSM8K (n=100)
PyTorch bf16 (reference, MPS)	60.0%	23%
LiteRT int8	58.3%	23%
LiteRT int4 (block-32)	53.3%	22%

int8 is statistically indistinguishable from bf16 on paired prompts. int4 costs ~7 points of instruction following — that is the price of the 89 MB saving. GSM8K is near floor for every configuration, as Liquid's own model card implies; no quantization changes it.

One recipe note: the obvious export-time int8 recipe (which also quantizes the conv layers) measured 53.3% on the same harness — 5 points below shipping int8. These bundles keep the convs float, matching what the LFM2.5-1.2B-JP and 2.6B conversions found for this architecture.

Usage

litert-lm run ./LFM2.5-230M_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."

# GPU
litert-lm run ./LFM2.5-230M_int8.litertlm --backend gpu --cache no --prompt "..."

The bundle embeds the model's own chat template (ChatML role markers, tool-calling branch included) and declares stops <|im_end|> and <|endoftext|>. KV budget is 4096 tokens with 11 prefill signatures (1–1024).

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine, ≥300 s rest before each GPU reading:

File	Backend	Prefill (256)	Decode	TTFT
int8	GPU	12810 tok/s	561 tok/s	0.022 s
int8	CPU	1727 tok/s	154 tok/s	0.155 s
int4	GPU	12862 tok/s	555 tok/s	0.022 s
int4	CPU	1613 tok/s	149 tok/s	0.166 s

Galaxy S26 (SM-S942Q, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown):

File	Backend	Prefill (205)	Decode	Peak RSS
int8	GPU (OpenCL)	3813–3815 tok/s	121.7–121.9 tok/s	444 MB
int8	CPU	1006–1140 tok/s	94.7–95.4 tok/s	636–706 MB
int4	GPU (OpenCL)	1101–2476 tok/s	41.6–43.6 tok/s	406 MB
int4	CPU	262–263 tok/s	50.7–53.9 tok/s	538 MB

The int4 GPU prefill range's low end is the cold shader-compile run. Note the ordering: at 230M, int8 decodes 2.8× faster than int4 on this GPU — the blockwise dequant overhead dominates at this size. The larger LFM2.5 models show the reverse.

iPhone 17 Pro (BenchmarkApp yardstick, 128-token prompt, cold runs, one reading per cell):

File	Backend	Prefill (128)	Decode	TTFT	Peak resident
int8	Metal GPU	3422 tok/s	143 tok/s	75 ms	610 MB
int8	CPU	1530 tok/s	87 tok/s	107 ms	941 MB
int4	Metal GPU	3036 tok/s	162 tok/s	71 ms	650 MB
int4	CPU	808 tok/s	94 tok/s	206 ms	791 MB

iPhone Metal is the one backend where int4 out-decodes int8 (162 vs 143 tok/s).

Conversion notes

Converted with litert-torch 0.9.3 / transformers 5.14.1 / ai-edge-quantizer, released wheels only (reproduction scripts: hf-to-litertlm).

• The upstream chat template does not run on the runtime's Jinja engine. LiquidAI's chat_template.jinja uses HF's {% generation %}/{% endgeneration %} training-mask markers (a parse error in minijinja) and message.get("content") (minijinja maps have no .get method). The embedded template strips the two markers and rewrites the two .get calls to plain indexing. Rendering is byte-identical to the original across user/system/multi-turn/tool-calling conversations, verified through HF apply_chat_template.
• KV budget 4096, not 4099. With the 1024-token prefill signature present, a 4099-token KV cache fails GPU engine creation at shader compile (measured on macOS WebGPU); 4096 — a multiple of the widest signature — compiles everywhere we tested.
• Convs stay float in the int8 file (see Accuracy). int8 is applied post-export to linears + embedding.
• ExecutorMetadata added post-export (the released exporter omits it for this state-carrying architecture; litert-lm ≥ 0.15 needs it to bind the 8 conv states and 12 KV buffers). Weights are byte-identical through the edit.
• Stops declared as token ids 7 (<|im_end|>) and 2 (<|endoftext|>); BOS <|startoftext|>. Tokenizer embedded as the upstream tokenizer.json.

License and changes

Distributed under the LFM Open License v1.0 (inherited from the base model; see the license link above). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and a render-equivalent repair of the chat template repackaged into the .litertlm bundle; runtime metadata (stops, ExecutorMetadata) added. This repository is a community conversion and is not affiliated with Liquid AI.