LFM2.5-1.2B-Instruct — LiteRT-LM

LiquidAI/LFM2.5-1.2B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14 / a recent AI Edge Gallery).

LFM2.5-1.2B-Instruct is Liquid AI's flagship on-device model: a hybrid architecture mixing gated short-convolution blocks with grouped-query attention. Per the base model card it outperforms Qwen3-1.7B and Gemma 3 1B on knowledge and instruction-following benchmarks at a smaller footprint.

File	Recipe	Size	GSM8K (n=100)
LFM2.5-1.2B-Instruct_int8.litertlm	int8 dynamic (linears + convs + embedding)	1.25 GB	81% (bf16 reference: 79%)
LFM2.5-1.2B-Instruct_int4.litertlm	int4 blockwise-32 + OCTAV linears, int8 embedding, convs float	736 MB	72%

Context (KV cache)	4096 max (see --max-num-tokens note)
Backend	CPU (the hybrid conv graph is not supported by current GPU delegates)
Template	bundled — full chat template incl. tool-list support + <think> channel declaration
Base model	LiquidAI/LFM2.5-1.2B-Instruct (LFM Open License v1.0)

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 1024, n=100, same harness for all rows:

Configuration	GSM8K
PyTorch bf16 (reference)	79%
LiteRT int8 (this repo)	81%
LiteRT int4-b32 OCTAV (this repo)	72%

The int8 file is at full parity with the PyTorch reference (+2pt, within noise). Both files pass an 8-question sanity gate (7/8, zero degenerate outputs). The int4 file trades −7pt for a 41% smaller download.

Usage

litert-lm run ./LFM2.5-1.2B-Instruct_int8.litertlm --prompt "Explain what a hybrid conv-attention LLM is in two sentences."

The bundle carries the tokenizer and the full chat template (ChatML-style <|im_start|>…<|im_end|> with tool-calling support). Generation stops cleanly at <|im_end|>.

--max-num-tokens tip: decode speed drops as the token budget grows (int8 decode: 101 tok/s at 1024 → 77 tok/s at 4096 on an M4 Max) — set it to the smallest value your use case needs (1024 is a good chat default; the file allows up to 4096).

Speed

litert-lm benchmark, CPU backend, Mac M4 Max, max-num-tokens 1024:

Variant	Prefill (256)	Prefill (1024)	Decode	TTFT
int8	402 tok/s	1603 tok/s	101 tok/s	0.65 s
int4	304 tok/s	—	91 tok/s	0.85 s

On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery): int8 decodes at ~19 tok/s and int4 at ~31 tok/s — on phone-class memory bandwidth the int4 file is ~1.7× faster as well as 41% smaller, so prefer int4 on mid-range devices unless you need the last few GSM8K points.

The hybrid short-conv blocks are what make a 1.2B decode at ~100 tok/s on CPU — faster than same-size pure-attention models. Long prompts prefill through the large (up to 1024-token) signatures at >1500 tok/s.

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat.

Conversion notes

Converted with released litert-torch 0.9.1 (which ships the lfm2 hybrid model support) and packaged for litert-lm ≥ 0.14, with one exporter patch: the stock LFM2 short-conv block saves its conv state from the last columns of the prefill chunk, which are padding whenever the prompt does not exactly fill a prefill signature — corrupting the first generated token of nearly every reply. The patch derives the valid length from the attention mask in-graph and gathers the conv state from the last valid columns instead; with it, engine output is token-identical to an exact per-token reference loop at every prompt length. Multi-length prefill signatures (1–1024) are exported so the runtime can pick tight chunks.

Quantization targets fully-connected + embedding ops (plus convs in the int8 file, quantized at export time); post-hoc int8 of the convs with ai-edge-quantizer breaks generation — quantize convs only at export time.

License and changes

Distributed under the LFM Open License v1.0 (see LICENSE, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.