LFM2.5-350M — LiteRT-LM

LiquidAI/LFM2.5-350M converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime. Tested on litert-lm 0.18.0 on a Mac; the phone rows ran the prebuilt Android binary named below. On litert-lm 0.17.1 the int8 file answered as if no question had been asked: the runtime prefilled 9 tokens for a one-question turn where 0.18.0 prefilled 16, the full turn (the int4 file was not tried on 0.17.1).

Liquid AI describes LFM2.5-350M as a general-purpose text-only model and recommends it for data extraction, structured outputs and tool use, not for knowledge-intensive tasks and programming. It has 350M parameters in 16 layers (10 double-gated convolution blocks + 6 grouped-query attention blocks), tied embeddings and a 65,536-token vocabulary; Liquid AI lists English, Arabic, Chinese, French, German, Japanese, Korean, Portuguese and Spanish and states a 32,768-token context (4096 in these files).

File	Recipe	Size	sha256
LFM2.5-350M_int8.litertlm	int8 dynamic on linears + int8 embedding, convs float	403.4 MB	6a78c7a2746386f8919476a34ef0ca40d596a5c44e979520fae8318da5cfb1f6
LFM2.5-350M_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 lm_head + embedding	247.7 MB	272b0a1ca55f8b39fe636927deffe4049feb93b5eda7c05a5dbbb3141d6d53f8

On IFEval-lite (n=120) the three scores are within three prompts of each other (table in Accuracy); int4 is 38.6 % smaller. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int8 GPU	0.68	2.94	5.77
int4 GPU	0.67	2.63	5.94
int8 CPU	2.38	3.77	8.13
int4 CPU	2.94	3.99	7.97

Which file is faster differs by device: int4 on the Galaxy S26 GPU, int8 on the Pixel 8a GPU, the two within 1 % on the Mac GPU. Not measured on iOS.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, greedy, one litert-lm run process each): int8 6/8 on CPU and on GPU (misses: "What is 8 times 7?" → "64"; "Which is larger: 0.9 or 0.11?" → "0.11 is larger."), int4 7/8 on CPU and on GPU (missing only the second: "0.11 is larger than 0.9."), bf16 (transformers) 7/8, missing the same question.

All 12 signatures of both files were fully delegated on the three GPUs measured (LITERT_WEBGPU on the Mac, LITERT_CL on both phones), and every phone leg answered "What is the capital of France? Answer in one sentence." correctly. A prompt-length check on CPU (41 prompt lengths, 19 and 21 to 60 tokens, a fresh engine each) returned the expected word at every length from both files; it was not run on GPU. In a three-turn conversation on CPU, int8 gave bf16's three replies word for word; int4 answered with a different word ("large") and completed all three turns.

Accuracy

IFEval-lite scores the first 120 rows of google/IFEval whose instruction types it checks mechanically: prompt-level strict, greedy; the litert-lm rows ran one litert-lm run process per prompt. The scores are comparable only within this table, not with published IFEval scores.

Configuration	Prompts passed (n=120)	Same file, Mac GPU (WebGPU on Metal)
bf16 reference (transformers, MPS)	81 (67.5 %)	—
int8, litert-lm CPU	82 (68.3 %)	86 (71.7 %)
int4, litert-lm CPU	79 (65.8 %)	78 (65.0 %)

Prompt by prompt, int8 on CPU failed 6 prompts that bf16 passed and passed 7 that bf16 failed; int4 failed 9 and passed 7. Between the CPU and Mac GPU runs of the same file, 8 (int8) and 13 (int4) prompts flipped, in both directions. On GSM8K (test questions 0–99, greedy, at most 1024 output tokens; bf16 in transformers, int8 and int4 on litert-lm CPU) bf16, int8 and int4 answered 2, 2 and 8 of 100 correctly with the harness's prompt and 7, 5 and 14 with a second wording of it; both wordings ask for step-by-step reasoning, and most bf16 and int8 replies were the final-answer line alone, with no working.

Usage

litert-lm run ./LFM2.5-350M_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./LFM2.5-350M_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm benchmark ./LFM2.5-350M_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu

Both run commands printed May 3, May 17; the int4 file answered with a list of the two dates, and no reply contained a chat marker such as <|im_end|>. Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model (358 MB for int8, 231 MB for int4 on CPU). For Android, see android-gpu.md.

The bundle's default sampler is temperature 0.1, top-k 50, top-p 1.0 (Liquid AI's recommended temperature and top_k; its recommended repetition_penalty 1.05 is not carried). run uses that sampler unless you pass --top-k, --top-p, --temperature or --seed; --top-k 1 gives greedy output.

The chat template is byte-identical to LiteRT-LM's models/lfm2/chat_template.jinja at commit 86a7e6d0, with its stops <|im_end|> and <|endoftext|> and start token <|startoftext|>. The metadata marks the model non-thinking (supports_thinking false) and keeps that configuration's thought channel (<think> / </think>).

Tool calling is declared (model type lfm2). The template lists tools in the system turn as List of tools: [ … ], the same text Liquid AI's template renders for one tool; after a tool call the two templates differ in two places (call arguments in double quotes, the tool result wrapped in [ … ]). On litert-lm 0.18.0 (Python API, CPU, greedy) both files returned a parsed get_weather call with {"city": "Paris"}, and with automatic tool calling the reply used the tool's result.

Engine reuse (int8 file, Python API, CPU, greedy, the same in 3 of 3 processes): on an Engine that had already served one conversation, 2 of the next conversation's 3 answers changed — "What is 17 times 23?" got 323, where a fresh Engine answered 391, the correct product — while both conversations still followed their system prompts. A related open issue is LiteRT-LM #3165. Open a new Engine for each conversation.

Performance

Mac M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations; GPU is WebGPU on Metal, after a 300 s rest. Init is the first iteration's engine init.

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	13556.05	423.73	0.0780	1.0924
int4 GPU	13965.02	425.85	0.0757	1.1679
int8 CPU	2200.63	134.00	0.4730	3.7308
int4 CPU	1195.41	123.11	0.8649	1.1431

Phones, Android 16: Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno) and Pixel 8a (Tensor G3), the same workload through the prebuilt litert_lm_advanced_main (sha256 adac974b…, 40,382,152 bytes) published 2026-09-18 in LiteRT-LM's public binary bucket (binaries/latest/android_arm64/litert_lm/) with --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true. Rates are the mean of iterations 2 and 3 (GPU: LITERT_CL). Init cold = the first process after the compiled-program caches were removed, warm = the benchmark process, which read those caches; peak = the runtime's Peak system ram usage line, the maximum of the 3 iterations.

Galaxy S26	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	5660.6	92.9	0.19	4.07 / 0.95	506.7
int4 GPU	4391.3	106.9	0.24	4.04 / 1.19	472.2
int8 CPU	1752.7	80.31	0.60	0.52 / 0.12	1274.0
int4 CPU	875.9	90.63	1.19	0.82 / 0.60	1120.9

The CPU rows ran with the clock lowering during them (lowered in 3 of 6 samples for int8 and 5 of 7 for int4, read every 2 s); decode fell across the three iterations (89.63 → 87.28 → 73.34 tok/s int8; 101.67 → 97.12 → 84.14 int4).

Pixel 8a	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	2410.6	47.9	0.445	11.56 / 1.56	939.2
int4 GPU	1245.9	50.06	0.84	6.08 / 1.83	872.2
int8 CPU	509.1	41.81	2.04	0.77 / 0.46	979.0
int4 CPU	342.1	51.41	3.02	1.46 / 1.25	836.0

No clock reduction was recorded in the other six phone runs (the Pixel 8a GPU clock could not be read by the benchmark harness; its GPU runs were checked on the CPU clock). Not run on the NPU.

Conversion notes

• Source and stack: LiquidAI/LFM2.5-350M at revision 9e6c6ccf; litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, ai-edge-litert 2.2.0, transformers 5.14.1, torch 2.13.0 (released wheels); packed by the litert-lm 0.18.0 CLI. lfm_work/convert_lfm25_350m.sh in hf-to-litertlm rebuilt both files with all four sections (metadata, state binding, tokenizer, model) byte-identical (the whole-file sha256 differs only by the packer's uuid and timestamp).
• Order: unquantized export (11 prefill signatures, 1024 down to 1, plus decode, over a 4096-position KV cache; 22 state buffers: 10 ShortConv states, 6 key and 6 value caches) → post-hoc int8 and int4 quantization → a scan of the int4 block scales (none of the 92 blockwise tensors has a zero or subnormal scale) → metadata repack with LiteRT-LM's lfm2 configuration.
• Template: Liquid AI's chat template fails at template creation on litert-lm 0.18.0 ({% generation %} markers), so the bundle carries LiteRT-LM's lfm2 template, which reads message content as a list of parts.

License and changes

Distributed under the LFM Open License v1.0 inherited from the base model (LICENSE); its section 5 conditions commercial use on an annual revenue below 10 million US dollars. Changes from the original work: exported to the LiteRT-LM format and quantized to int8 and int4 as described above; chat template and stop tokens taken from LiteRT-LM's models/lfm2 configuration, with max_num_tokens changed from 32768 to 4096 and supports_thinking from true to false. This conversion was made and is published independently of Liquid AI.