Falcon-H1-Tiny-R-0.6B — LiteRT-LM

tiiuae/Falcon-H1-Tiny-R-0.6B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.16. Sibling of the Falcon-H1-0.5B/1.5B/1.5B-Deep/3B-Instruct conversions — same hybrid-cache patch, plus reasoning-specific packaging.

Falcon-H1-Tiny-R is a reasoning model: it opens a <think>…</think> block on its own before answering. The bundle declares the thinking channel, so the runtime separates reasoning from the answer and --thinking-budget works. All 44 layers run a grouped-query attention branch and a Mamba2 selective-scan branch in parallel; each layer carries KV plus constant-size conv/SSM state.

File	Recipe	Size
Falcon-H1-Tiny-R-0.6B_int8.litertlm	int8 dynamic on linears (FC only); embedding table externalized and kept float; convs and the scan stay float; fp32 activations declared for GPU; thought channel declared	873 MB

Why the embedding stays float: on this checkpoint an int8 embedding table (even per-row) measurably destabilizes reasoning — runaway thinking and greedy flips that the FC-only recipe does not show. The embedder is split into its own CPU-side section so the decoder graph stays fully GPU-delegable.

Correctness

• GSM8K (100 questions, greedy, 2048-token budget, answers scored after </think>): this int8 bundle 76/100 vs the bf16 PyTorch model 71/100 on the identical prompt stream and protocol — per-question flips run in both directions (64 both-correct), i.e. quantization noise, not degradation.
• 8-question sanity gate (litert-lm 0.16.0, 2048-token budget): Mac CPU 6/8, Mac GPU 7/8, iPhone CPU 6/8, iPhone Metal 6/8 — every reasoning/logic/math item passes on every backend; the misses are two fact-recall items ("capital of Japan", "thank you in French") that the bf16 model also flips under one-token prompt perturbations at this scale.
• Thinking channel: verified on Mac CPU/GPU and iPhone — reasoning arrives on the thought channel (not inline), and a --thinking-budget 16 run cuts the reasoning at exactly 16 tokens.
• Prompt-stream fidelity: the bundle's template + tokenizer reproduce the upstream apply_chat_template token stream byte-for-byte from BOS on (measured; the start-token variant costs a correct answer at this size).

Usage

litert-lm run ./Falcon-H1-Tiny-R-0.6B_int8.litertlm --prompt "What is 17 + 25?"

# GPU
litert-lm run ./Falcon-H1-Tiny-R-0.6B_int8.litertlm --backend gpu --cache no --prompt "..."

# cap the reasoning
litert-lm run ./Falcon-H1-Tiny-R-0.6B_int8.litertlm --thinking-budget 256 --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. Stop tokens are <|end_of_text|> (11) and <|im_end|> (228), from the upstream generation_config.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	2216 tok/s	97.8 tok/s	0.13 s
CPU	381 tok/s	47.8 tok/s	0.69 s

On device (cold start, single runs, 148-token composite prompt, quality harness):

Device	Backend	Prefill	Decode	TTFT (cold)	Peak memory
iPhone 17 Pro	GPU (Metal)	259.8 tok/s	22.8 tok/s	12.9 s	3.01 GB
iPhone 17 Pro	CPU	223.9 tok/s	29.4 tok/s	8.8 s	0.82 GB

Honest notes:

• CPU decodes faster than Metal at this size (29.4 vs 22.8 tok/s) — GPU setup/dispatch overhead dominates a 0.6B model; the GPU row exists for completeness and for devices where the CPU is busy.
• The cold TTFT figures include first-launch backend initialization; warm TTFT is far lower (see the Mac rows).
• GPU runs with fp32 activations (declared in the bundle) — expect a corresponding memory multiple over CPU.
• Multi-turn fact recall is weak at this size — turn-3 "what did I tell you earlier" questions fail on the bf16 model too. Treat it as a single-turn reasoner.
• Reasoning length is prompt-sensitive: on nonsense/filler prompts the model can think past any budget without closing (also reproduced at bf16). Give it real questions, or set --thinking-budget.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm falcon_h1_work/):

• Composite hybrid cache layer: every layer holds KV + conv + recurrent state at one layer index (176 state buffers at 44 layers); the runtime binds states by tensor name.
• Folded selective scan: the Mamba2 scan re-expressed as batched matmuls (rank ≤ 4, no BROADCAST_TO, no int64 index math) — fully GPU-delegable.
• Externalized embedder: the token-embedding lookup is its own CPU-side section, so the float table never blocks GPU delegation of the decoder graph.
• Prefill-pad guard from position monotonicity: the externalized decoder graph carries no token ids, so pad positions are identified by non-increasing position ids and made exact identity steps for the SSM (without this, partially-filled prefill chunks corrupt the recurrent state on CPU).
• Reasoning packaging: thought channel (<think>/</think>) declared in LlmMetadata.channels; no start-token (the template carries the BOS literally, matching the upstream stream byte-for-byte); both upstream stop tokens.
• Quantization: post-hoc dynamic int8 over FC only; embedding, convs and the scan stay float.

License and changes

Distributed under the Falcon LLM License (inherited from the base model — see the license link). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer repackaged unmodified; chat template repackaged with the {{bos_token}} expression replaced by the literal token (rendering-equivalent). This repository is a community conversion and is not affiliated with TII.