Falcon-H1-0.5B-Instruct — LiteRT-LM

tiiuae/Falcon-H1-0.5B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Falcon-H1 in LiteRT form, and the first fully-hybrid (parallel attention + Mamba2 SSM in every layer) architecture served by the released runtime — including on the GPU.

Falcon-H1 is TII's hybrid design: every one of the 36 layers runs a grouped-query attention branch and a Mamba2 selective-scan branch in parallel on the same input and sums them. Each layer therefore carries both a KV cache and constant-size conv + SSM recurrent state.

File	Recipe	Size
Falcon-H1-0.5B-Instruct_int8.litertlm	int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU	650 MB

Correctness

• Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 8 decode positions — max|logit diff| 7.6e-05, correlation 1.000000, top-1 and top-5 identical at every position.
• 8-question sanity gate: int8 = float = GPU — all three answer 6/8 with near-verbatim identical text; the two misses ("merci" and 8×7) are the 0.5B model's own level (the float graph misses them the same way), not conversion or quantization damage.
• Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length, 12–60 tokens) — 41/41 clean.
• Devices: iPhone 17 Pro (Metal) runs the composite quality probe at 7/8; Pixel 8a (OpenCL) delegates every subgraph fully (e.g. 5366/5366, zero rejections) with correct output.

Usage

litert-lm run ./Falcon-H1-0.5B-Instruct_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Falcon-H1-0.5B-Instruct_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock ChatML-style Falcon-H1 chat template.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	2650 tok/s	127.5 tok/s	0.10 s
CPU	473 tok/s	59.0 tok/s	0.56 s

On device (cold start, single runs, 145-token composite prompt, quality harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	365.9 tok/s	52.7 tok/s	0.46 s	2.50 GB
iPhone 17 Pro	CPU	332.3 tok/s	36.2 tok/s	0.48 s	0.66 GB

Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, OpenCL, default 19-token prompt): TTFT 0.93 s, prefill 23.6 tok/s, decode 11.8 tok/s, full delegation.

Honest notes:

• GPU runs with fp32 activations (declared in the bundle) — that is the GPU memory multiple above (2.50 GB vs 0.66 GB on iPhone).
• At 0.5B the model itself is weak at arithmetic and non-English trivia; int8 adds borderline greedy flips on exactly those items (e.g. 8×7 differs between backends). Everything stays coherent — there is no degeneracy.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm falcon_h1_work/):

• Composite hybrid cache layer: every layer holds KV + conv + recurrent state at ONE layer index — a cache layer class that is full-attention and Mamba2 at the same time (the runtime binds states by tensor name, so co-residency is just packaging).
• Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
• Falcon-specific wiring: the µP multiplier vector (mup_vector, a non-persistent model-level buffer) and ssm_in_multiplier are preserved in the traced scan; the exporter's timestamp-index kwargs are re-injected at the attention layer (FalconH1's layer loop drops kwargs).
• Prefill-pad guard: the runtime runs partially-filled prefill chunks; pad positions are made exact identity steps for the SSM and the stored conv window is gathered at the last valid column.
• Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.

License and changes

Distributed under the Falcon LLM License (inherited from the base model — see the license link). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified. This repository is a community conversion and is not affiliated with TII.