Spark-X2.6-4B — LiteRT-LM

This repository is a conversion of XHToken/Spark-X2.6-4B to LiteRT-LM (.litertlm), made with litert-torch 0.9.3 for LiteRT-LM 0.19.0: on a Mac Studio (Apple M4 Max) GPU the int4 file decodes at 91.2 tok/s (1,024-token prompt, 256 generated tokens, 2026-10-31).

Spark-X2.6-4B is the SparkLLM Team's 4.11B-parameter dense decoder, the first model of the Spark-X2.6 series. Its model card says it builds on Spark-X2.5-4B, keeps its architecture and gains its capability from improved post-training. That architecture has 36 layers, hidden size 2560, 16 query and 4 KV heads of 256, three sliding-window (512) layers per full-attention layer, per-head sigmoid gates on the attention output, exact-GELU gated MLPs and a 131,072-entry tied vocabulary. It is a reasoning model: it thinks inside <think>…</think> before it answers. These files carry that: the prompt template opens the thought the way the vendor's chat template does, and the files declare a thought channel, so LiteRT-LM keeps the reasoning out of the answer. Apache-2.0.

File	Recipe	Size
Spark-X2.6-4B_int4.litertlm	int4 blockwise-128 + OCTAV on the linear layers, int8 embedding table in its own section	2483 MB
Spark-X2.6-4B_int8.litertlm	int8 (dynamic, per channel) on every linear layer and the embedding	4238 MB

Both files declare fp32 GPU activations. The int4 file also uses LiteRT-LM's GPU composite ops for the KV-cache writes and the attention products, with its weights unchanged (see Conversion notes); the int8 file keeps the graph as exported. The numbers for these files were measured with LiteRT-LM 0.19.0; earlier releases were not tested with these files.

On a GPU (a Mac, for example), use Spark-X2.6-4B_int4.litertlm: it decodes 91.2 tok/s on the Mac GPU, the int8 file 49.8, and scores GSM8K 89 against the bf16 model's 90. On the Mac GPU the int8 file scores 93 but decodes 45 % slower. On a CPU, use Spark-X2.6-4B_int8.litertlm: on the CPU the int4 file gave an empty third answer in our 3-turn conversation check in 3 of 8 runs (4 processes × 2 modes, with and without the runtime's channel filtering; the same check passes on the GPU), and the int8 file scores GSM8K 90 on the CPU (bf16 90) and passes the check (one process, both modes). On a Galaxy S26, use the int8 file on the CPU (5.3 tok/s) and stream long replies (see Known runtime issues); we have no GPU row for this model on that phone (see Performance).

Correctness

• 8 short questions with one right answer each (two arithmetic questions, a capital, a translation, a decimal comparison, an opposite, the days of a week and a rhyme; Mac, LiteRT-LM 0.19.0 Python API, one process per question, greedy, thinking on, 2,048 output tokens): int4 CPU 8/8, GPU 8/8; int8 CPU 8/8, GPU 8/8. Every thought closed, every answer was a clean final answer.
• On phones, the same 8 questions with the same settings (the LiteRT-LM 0.19.0 Android wheel from PyPI): Galaxy S26 CPU, int8: 8/8, each prompt rendered to the same bytes as on the Mac.
• Multi-turn (3 turns, greedy: plant a fact, do arithmetic, recall the fact; with and without the runtime's channel filtering): int4 passes on the GPU. On the CPU the int4 file passed both modes in its first process; in each of 3 more fresh processes the mode without channel filtering answered turns 1 and 2 right and then gave an empty third answer right after its thought (the bf16 model's answer at that point is more than 17 logits ahead of every stop token, so this is not a near tie in the model). int8 passes on the GPU and on the CPU (one process each, both modes).
• Tokenizer: the tokenizer section is the upstream tokenizer.json, byte for byte. With these bytes LiteRT-LM encodes 234 of 234 probe rows (every added token alone and mid-string, a role-marker turn, Latin-1 / Extended-A, emoji, CJK, digits, whitespace) to the same ids as the tokenizers library.
• Chat template: for system, user and assistant messages the files' template renders byte-identically to the vendor's chat_template.jinja: 54 renders (three conversation shapes × generation prompt on / off × thinking unset / on / off × reasoning_effort unset / high / low), in transformers and in LiteRT-LM 0.19.0.

Accuracy — GSM8K

GSM8K, the first 100 test questions, greedy, thinking on, a 0-shot chain-of-thought prompt that asks for the answer as #### <number>, 3,584 output tokens (the files' KV cache holds 4,096), one conversation per question. The LiteRT-LM rows ran on a Mac (LiteRT-LM 0.19.0 Python API), the reference with transformers on the Mac (MPS) and the vendor chat template. The answer is read from the text after the thought, or from the thought when the model did not finish within the 3,584 tokens; the same rule for every row.

Configuration	GSM8K
PyTorch bf16 (reference)	90 (7 unfinished)
LiteRT-LM int4, GPU	89 (4 unfinished)
LiteRT-LM int8, GPU	93 (6 unfinished)
LiteRT-LM int8, CPU	90 (6 unfinished)

Question by question against bf16: int4 84 both right, 6 right only in bf16, 5 right only in int4; int8 (GPU) 89 / 1 / 4; int8 (CPU) 86 / 4 / 4.

Usage

pip install litert-lm==0.19.0
litert-lm run ./Spark-X2.6-4B_int4.litertlm --backend gpu --thinking true --prompt "What is the capital of France? Answer in one word."
litert-lm run ./Spark-X2.6-4B_int8.litertlm --backend cpu --thinking true --prompt "What is the capital of France? Answer in one word."

Python, with the vendor's thinking levels (reasoning_effort high is the default, low asks for a shorter thought, and thinking off renders the vendor's no-think prompt):

import litert_lm

engine = litert_lm.Engine("Spark-X2.6-4B_int4.litertlm", backend=litert_lm.Backend.GPU(), max_num_tokens=4096)
conversation = engine.create_conversation(extra_context={"reasoning_effort": "low"})
reply = conversation.send_message("How many prime numbers are smaller than 30?", max_output_tokens=3584)
print(reply["channels"]["thought"])
print(reply["content"][0]["text"])

• Give it a generous output budget (3,584 tokens for math). The model thinks before it answers; cut off mid-thought it gives no final answer.
• litert-lm run writes its compile caches next to the model file by default: 2.08 GB in two files beside the int4 file after the GPU command above (a 2.06 GB weight cache and a 24 MB program cache), 4.13 GB in one file beside the int8 file after the CPU one. --cache no turns that off.
• create_conversation(thinking_config=litert_lm.ThinkingConfig(enable_thinking=False)) renders the vendor's no-think prompt (<｜start▁of▁sentence｜><|Bot|></think>).
• The template is the vendor's plain-chat part: a default system block (<|System|>, "you are a helpful assistant.", the thinking_effort instruction while thinking is on, then your system prompt), every message wrapped in <｜start▁of▁sentence｜> … <｜end▁of▁sentence｜>, generation prompt <｜start▁of▁sentence｜><|Bot|><think>; stop token <｜end▁of▁sentence｜>. Tool-call formatting is not carried. KV cache 4,096 tokens (the original's 1M context does not apply here), six prefill sizes (1024, 256, 64, 16, 4, 1) — six rather than eleven because every exported prefill size takes engine memory whether or not it is called.
• The vendor's recommended sampling is temperature 1.0 / top-p 0.95; the numbers on this card are greedy.

Performance

Every row: prefill of 1,024 tokens, then 256 decoded tokens, with the context set to 2,048 tokens; 3 runs per process, at least two processes per file, and the value is the mean over processes. LiteRT-LM 0.19.0 from PyPI.

Mac Studio (Apple M4 Max, 128 GB, macOS 27.0), litert-lm benchmark <file> -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --backend <cpu|gpu> --cache no (one warm-up run, then the mean of 3):

File	Backend	Prefill	Decode	TTFT	Init
int4	GPU (WebGPU)	1,200 tok/s	91.2 tok/s	0.86 s	3.2 s
int4	CPU	128 tok/s	17.4 tok/s	8.07 s	5.2 s
int8	GPU (WebGPU)	1,100 tok/s	49.8 tok/s	0.95 s	13.0 s
int8	CPU	324 tok/s	17.6 tok/s	3.35 s	42.8 s

Init is the Init time the CLI prints; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

Galaxy S26 (SM-S942Q, Snapdragon, Adreno; Android 16), the litert-lm-api 0.19.0 Android wheel from PyPI, Benchmark(model, backend, prefill_tokens=1024, decode_tokens=256, max_num_tokens=2048), 3 runs per process on one engine; a process's value is the mean of runs 2 and 3 (run 1 is the first use); CPU runs started from an unthrottled phone:

File	Backend	Prefill	Decode	TTFT	Processes
int8	CPU	89 tok/s	5.3 tok/s	11.8 s	4

No Galaxy S26 GPU row: our test stops a run at the first reading of MemAvailable (the kernel's estimate, in /proc/meminfo, of the memory available to new work without swapping) under 1,000,000 KiB (read every 0.25 s). The int8 file's GPU engine build (context 2,048) was stopped there twice, at 987,700 and 959,672 KiB; the int4 file before the GPU graph rewrite, at the 4,096-token context these files are exported with, was stopped four times, at 939,996 to 988,316 KiB (at context 2,048 it ran, 8.6 tok/s decode); the int4 file in this repository was not run on that GPU. Pixel 8a (8 GB of RAM): the int8 file's CPU benchmark run did not finish within the runtime's 10-minute limit (DEADLINE_EXCEEDED; the phone was swapping). iPhone 18 Pro: our app could not create the int8 file's engine (its XNNPACK cache file could not be memory-mapped, and on Metal "Could not mmap file"); the app is signed without the increased-memory-limit entitlement, and whether that is the cause was not tested. The int4 file was not run on either phone.

No NPU file. We also compiled this model for the Galaxy S26 NPU (Qualcomm HTP, ahead of time for SM8850, QAIRT 2.47): the compiled model's context did not load on the phone (the runtime logged a context size estimate of 3,820,573,952 bytes and error 1002, QNN's memory-allocation error; QAIRT 2.47's HTP documentation gives 3.75 GB of usable address space per DSP session), so there is no NPU file.

Conversion notes

Converted with litert-torch 0.9.3 / transformers 5.14.1 / ai-edge-quantizer 0.8.0, released wheels; files packed with the litert-lm 0.19.0 CLI (reproduction scripts: hf-to-litertlm).

• The vendor modeling code is patched for export, not re-implemented (the patch of the Spark-X2.5 conversion; X2.6 ships the same modeling_spark.py). modeling_spark.py computes attention through its own eager function and ignores config._attn_implementation; the export copy dispatches through the registered attention interface (so litert-torch's transposed KV cache is used), threads the per-call kwargs, declares the attention-backend capability flags, and applies the per-head sigmoid output gate in the interface's layout. In eager mode the patched file is bit-identical to the vendor file (max |Δlogit| 0.0 on 24 random tokens).
• The think opener is pre-filled by the template and the thought channel is declared in the metadata. No start token in the metadata: the tokenizer declares <｜start▁of▁sentence｜> as BOS but never prepends it (add_bos_token: false), and the template writes its own.
• int4 keeps the embedding table in its own section. The vocabulary is tied; asking int4 for the output layer and int8 for the embedding would make the quantizer copy the 131,072-row table once per signature. The int8 file needs no split.
• GPU graph (int4 file). After the export, the int4 file's KV-cache writes and attention products of every signature are rewritten as LiteRT-LM's GPU composite ops (odml.cache_update, odml.runtime_bmm) with tools/gpu_graph/gpu_graph_retrofit.py from hf-to-litertlm (--decomp exporter --mask add_bcast, plus the band-mask extension the sliding-window layers need, dory_work/retrofit_gpu.py in the same repository); every weight buffer is byte-equal to the export's. Mac GPU decode against the same file without the rewrite: ×1.72 (53 → 91 tok/s); CPU speed did not change measurably; GSM8K gave the same verdict on every one of the 100 questions with and without the rewrite. The int8 file keeps the exported graph: it is the file we recommend for the CPU, and the same rewrite of the int8 weights changed nothing measurable on the Mac CPU (18.1 vs 17.6 tok/s; the two processes of each file, 18.4 / 17.8 and 17.8 / 17.4, overlap).
• GPU activations are declared fp32 in the files (prefer_activation_type = "fp32"). LiteRT-LM runs the text decoder with fp16 activations on the GPU by default, and on this 36-layer decoder that costs reasoning accuracy: the same int8 weights scored GSM8K 79 with fp16 GPU activations forced and 93 with fp32 declared (bf16 90). fp32 activations take more GPU memory: on the Galaxy S26 the int8 file's weights (context 2,048) built an fp16-activation GPU engine at 6.94–6.96 GB of the GPU driver's page pool (and was then stopped by the same memory limit within its first runs, twice), while the fp32 build passed 7.46–7.70 GB and was stopped by our memory limit before it finished, twice (see Performance).
• Tokenizer: the upstream tokenizer.json, byte for byte (the exporter's own copy marks <unk> as special, which the upstream file does not; replaced).
• Quantization at export time (int4 block scales floored at 2^-14; none needed it here); no other post-processing of the weights.

Known runtime issues

On litert-lm 0.18.1 and 0.19.0, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (a fixed timeout with no setting), and the generation it leaves running can crash the process. For long thinking on a slow backend, lower max_output_tokens or use the streaming send_message_async, which has no such limit. On the Galaxy S26 CPU a full 3,584-token reply of this model takes longer than 10 minutes (decode 5.3 tok/s), so stream there.

Later answers in a conversation can differ from the answer the model gives to the same messages in one prompt. With thinking off, from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because Spark-X2.6-4B_int4.litertlm holds the embedding table in its own section and the decoder takes embeddings (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). With thinking on, the runtime rebuilds the cache from the re-rendered history at each turn, so the second turn is free of it when the conversation opens with a user message; the same source shows a zero embedding at another token from the third turn on (not measured). Spark-X2.6-4B_int8.litertlm has no embedder section (the decoder looks the tokens up itself) and is not affected. One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.

An answer whose last punctuation mark is a token of its own, such as a . or ? after a word, can arrive without that mark. On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1) the cause is this: these files carry <｜end▁of▁sentence｜> with each of 13 characters in front (12 punctuation marks and a space) as stop sequences next to <｜end▁of▁sentence｜> alone, so the runtime holds the mark back to see whether <｜end▁of▁sentence｜> follows, and when the answer ends there the held mark is never written. From LiteRT-LM 0.17.0 on (read in the source of 0.17.0 to 0.18.1) Spark-X2.6-4B_int4.litertlm and Spark-X2.6-4B_int8.litertlm still drop all 13: the mark and <｜end▁of▁sentence｜> complete two stop sequences at the same token, <｜end▁of▁sentence｜> alone and the one with the mark, the runtime records the one that comes later in the file's stop list (for <｜end▁of▁sentence｜> alone, its first entry when the file lists it twice), and it writes the held mark only when that is <｜end▁of▁sentence｜> alone. Not measured on these files: this follows from the runtime's source read with these files' stop list and tokenizer; the drop on releases before 0.17.0 was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle with punctuated stop sequences.

License and changes

Distributed under Apache-2.0 (inherited from the base model; the upstream LICENSE file is included). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; in the int4 file, the attention's KV-cache writes and products rewritten as LiteRT-LM GPU composite ops (weights unchanged); fp32 GPU activations declared; tokenizer and chat template repackaged into the .litertlm files (thinking pre-fill and thought-channel metadata as described); tool-calling template branches not carried. No training or fine-tuning. This conversion was made independently of the SparkLLM Team.