Spark-X2.5-4B-MLX-6bit

A 6-bit MLX conversion of Spark-X2.5-4B, a compact agentic model from the SparkLLM team with a native 1M-token context window. At 6 bits the quantization loss disappears into the noise floor — measurably indistinguishable from bf16 on perplexity — at 3.1 GB on disk, ~3.2 GB GPU memory at runtime.

Same treatment as the 4-bit build, just gentler: everything quantized to 6-bit including the tied embedding table, with the head-wise attention output gates (g_proj, ~187K params) kept in BF16 per the upstream guidance.

Quantization details

Format	MLX affine, 6-bit, group size 64
Effective bits per weight	6.504
Model size	3.1 GB (from 8.2 GB bf16)
Kept at BF16	self_attn.g_proj gates + RMSNorm weights
Embeddings	6-bit (tied lm_head)

Converted with mlx-lm 0.31.3 and the official Spark-MLX-LLM architecture support, orchestrated by Hermes Agent.

Loading

spark2_5 isn't in the latest mlx-lm PyPI release yet (it has landed on mlx-lm's GitHub main), so this repo ships the architecture file (spark2_5.py) and loads it via mlx-lm's model_file mechanism. Requires mlx-lm ≥ 0.31. No extra packages needed:

pip install -U mlx-lm

python3 -m mlx_lm.generate \
  --model hermitdave/Spark-X2.5-4B-MLX-6bit \
  --prompt "Explain why hybrid attention saves memory at long context." \
  --max-tokens 512

from mlx_lm import load, generate

model, tokenizer = load("hermitdave/Spark-X2.5-4B-MLX-6bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is the capital of Anhui Province?"}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Alternatively, install Spark-MLX-LLM for their spark-mlx-* CLI wrappers and tool-call parsing.

Benchmarks (measured, not copied)

Run on an M3 Max (64 GB) by Hermes Agent at conversion time. Perplexity on ~24.6k tokens of Pride and Prejudice (Gutenberg), 1024-token windows:

	bf16 base	this 6-bit	4-bit
Perplexity ↓	14.45 ± 0.27	14.56 ± 0.27	16.33 ± 0.31
Disk size	8.2 GB	3.1 GB	2.2 GB
Runtime GPU memory	~8.3 GB	3.15 GB	2.26 GB
Decode speed	—	20 tok/s	21 tok/s

The 6-bit perplexity sits well within one standard error of the bf16 baseline — no measurable quality loss. The interesting cliff in this family is between 6-bit and 4-bit: dropping to 4-bit costs +13% perplexity, while 6-bit costs nothing measurable for 0.9 GB more.

Quality smoke tests all passed at temp 0: 53 × 42 = 2226 (matches the vendor's reference output), train speed 80 km/h, capital of Anhui → Hefei, correct Chinese-language arithmetic, and a well-formed get_weather(city="Paris", unit="celsius") tool call.

Architecture

Spark-X2.5 uses a hybrid attention pattern — 3 sliding-window layers followed by 1 full-attention layer, repeating:

• 36 layers (27 sliding + 9 full), window 512
• Hidden 2560, 16 attention heads / 4 KV heads, head dim 256
• GQA + partial RoPE on full-attention layers (25% rotary, θ=5M); full RoPE θ=10k on sliding layers
• Head-wise sigmoid attention output gates
• Parallel GELU MLP, intermediate 10240
• Vocab 131,072, tied embeddings
• Native context: 1,048,576 tokens

Thinking mode is on by default via the chat template; pass enable_thinking=False to apply_chat_template to disable. Recommended sampling: temperature 1.0, top_p 0.95.

License

Apache 2.0, same as upstream. The bundled spark2_5.py is derived from Spark-MLX-LLM (Apache 2.0).

Attribution

• Upstream model: XHToken/Spark-X2.5-4B by the SparkLLM team
• Architecture support: Spark-MLX-LLM
• Conversion & validation: Hermes Agent by Nous Research, using mlx-lm 0.31.3

Citation

@misc{sparkx2.5,
    title  = {Spark-X2.5 4B&1.7B: Pushing the Limits of Agentic Capabilities in On-Device Models},
    author = {SparkLLM Team},
    year   = {2026}
}