sermonis-ai/Kestrel-4B-Coder-oQ4e-mtp

Kestrel-4B-Coder — MLX oQ4e (BF16), MTP + vision

This is a 4-bit MLX quantization of Jackrong/Kestrel-4B-Coder, a coding and agent-focused fine-tune of Qwen3.5-4B. It was produced with oMLX's oQ4e streaming quantizer and preserves both the vision tower and the integrated multi-token-prediction (MTP) head.

A sibling FP16 build is available as Kestrel-4B-Coder-oQ4e-fp16-mtp.

Model details

Field	Value
Base model	Jackrong/Kestrel-4B-Coder @ 46fdfa2c75949c6e3abd4c62d7a1c079ffc18245
Upstream lineage	Qwen/Qwen3.5-4B → unsloth/Qwen3.5-4B → Kestrel
Architecture	qwen3_5 VLM (Qwen3_5ForConditionalGeneration)
Quantization	oQ4e — oQ level 4, enhanced imatrix, group_size=64, affine
Storage dtype	bfloat16 (quantized weights packed as U32)
MTP	preserved — 1 layer, 29 tensors under language_model.mtp.*
Vision tower	preserved — 297 tensors, BF16, not quantized
Tensors	1250 total (953 language + 297 vision); 0 lm_head
Size	3,265,548,624 bytes (3.27 GB)
Imatrix	oqe_code_multilingual, 128 samples × 512 tokens, 256 entries
Runtime	oMLX ≥ 0.7.1 with MLX 0.32.4

Quantization recipe

quantize_oq_streaming(
    model_path="Kestrel-4B-Coder",
    output_path="Kestrel-4B-Coder-oQ4e-mtp",
    oq_level=4, group_size=64,
    text_only=False, dtype="bfloat16",
    preserve_mtp=True, enhanced=True,
    imatrix_num_samples=128, imatrix_seq_length=512,
    auto_proxy_sensitivity=True, imatrix_reuse_cache=True,
    stream_calibration=None,
)

text_only=False keeps the vision tower at source precision; preserve_mtp=True keeps the MTP head and its config fields. The imatrix cache was freshly collected for this build (cache_reused=false).

Verification report

All tests below were run against this artifact after quantization, on 2026-10-26.

Environment: Apple M2 Max, 96 GiB unified memory; macOS (Darwin 25.6.0); oMLX 0.7.1; MLX 0.32.4; mlx-lm 0.31.4.dev132; mlx-vlm 0.7.8; transformers 5.17.0; Python 3.11.17.

#	Test	Result	Measured
T1	Source integrity	PASS	738 tensors; both shard SHA-256 match the Hub (9.32 GB)
T2	Pre-quant estimate	PASS	5.61 bpw, 3.267 GB expected
T3	Structural audit	PASS	1250 tensors; 297 vision (all BF16, zero .scales/.biases); 29 MTP; 0 lm_head; U32 256 / BF16 994
T4	Config audit	PASS	mtp_num_hidden_layers=1 (root and text), unsloth_fixed_mtp=true, 4-bit/64/affine, 92 per-layer boosts
T5	Patched load (mlx_vlm + oMLX patch)	PASS	language_model.mtp attached with 1 layer; MTP norm matches on-disk tensor; vision 333.51M params
T6	Text smoke, 5 coding prompts × {greedy, shipping}	PASS	Coherent, EOS-terminated, no truncation artifacts
T7	Tool-call JSON	PASS	Output parsed as valid JSON with expected name/arguments
T8	Vision smoke, 2 images	PASS	Identified the “HumanEval and HumanEval+ / Coding Quality” comparison table and the “BenchLocal / HermesAgent-20” horizontal grouped bar chart
T9	Long answer (512 tokens)	PASS	227 words; max repeated 15-gram = 1
T10	MTP throughput	PASS	22.01 vs 20.22 tok/s mean (+8.9%); 21.92 vs 18.28 median (+19.9%)
T11	Perplexity vs source BF16	PASS	1.8617 vs 1.8579 (+0.202%)
T12	Next-token KL vs source BF16	PASS	0.01579 nats mean

MTP throughput detail

Isolated omlx serve on port 8123, --no-cache, thinking disabled, temperature=0, max_tokens=256, 3 prompts × 3 measured runs per configuration (warmup discarded):

Configuration	Mean tok/s	Median tok/s	Mean completion tokens
MTP off	20.22	18.28	138.0
MTP on	22.01	21.92	137.4

Across 22 MTP-on decode cycles observed during the benchmark (both variants), the MTP runtime reported mean draft acceptance 74.7% and 1.80 tokens/cycle. Throughput variance is high; the ship gate was ≥1.05× mean, which both variants clear.

PPL/KL detail

Five held-out code snippets, 422 predicted tokens, compared against the unquantized source loaded sequentially:

Metric	Source BF16	This build
Perplexity	1.8579	1.8617
PPL delta	—	+0.202%
Mean next-token KL	—	0.01579 nats

How to run

oMLX discovers models as subdirectories of --model-dir:

omlx serve --model-dir /path/to/models --port 8000

The model id is the directory name: Kestrel-4B-Coder-oQ4e-mtp.

Enable MTP in ~/.omlx/model_settings.json:

{
  "version": 1,
  "models": {
    "Kestrel-4B-Coder-oQ4e-mtp": {
      "mtp_enabled": true,
      "mtp_adaptive_max_depth": 2,
      "mtp_fixed_depth": null,
      "enable_thinking": false,
      "temperature": 0.2,
      "top_p": 0.95,
      "top_k": 20
    }
  }
}

Stock mlx_vlm silently strips mtp.* on load, and mlx_lm cannot load the vision tower. MTP plus vision requires oMLX ≥ 0.7.1 with its Qwen3.5 VLM runtime patch.

Recommended sampling

• temperature=0.2, top_p=0.95, top_k=20
• MTP draft depth 2
• If long answers fall into repetition loops, retry with temperature=1.0, top_p=0.95, top_k=20

Caveats

• This is a quantized derivative, not a new base model. Quality claims are limited to the tests above; benchmark scores from the source model card were not re-run here.
• The BF16 build stores vision at BF16. The FP16 sibling stores vision/audio at float32 by design in oMLX, making it larger (~3.93 GB).
• transformers 5.17 emits a pre-tokenizer regex warning for this tokenizer. The tokenizer is unchanged from the source, so behavior matches the upstream model.
• The MTP benchmark is a small-sample local measurement; absolute tok/s is hardware- and context-dependent.

Acknowledgments and attribution

This build is entirely derivative. Credit for the foundational work:

• Jackrong — author of Kestrel-4B-Coder: the source weights, model card, evaluations, and sampling guidance. Thank you for the foundational work this conversion is built on.
• Qwen Team (Alibaba) — Qwen3.5-4B base architecture, tokenizer, vision tower, and MTP design; Unsloth — the fine-tuning framework and merged checkpoint (including the MTP/NextN weights and unsloth_fixed_mtp fix).
• Kyle Hessling — collaboration and support on Kestrel, per the source card.
• NVIDIA — Nemotron data ecosystem used upstream.
• jundot / oMLX — the oQ/oQe streaming quantizer, oQe imatrix calibration, Qwen3.5 VLM runtime, MTP patches, and Apple-Silicon kernels used to produce and run this artifact.
• Apple MLX, mlx-lm, and mlx-vlm — the runtime and model implementations.
• Hugging Face — Hub, transformers, huggingface_hub, and safetensors.
• Calibration data — oMLX’s oQe Balanced Code + Multilingual + Tool/Reasoning profile, built on public code and multilingual corpora.
• EvalPlus, llama.cpp, LM Studio, and BenchLocal — evaluation tooling referenced by the source card.
• mlx-works/Qwopus3.5-4B-Coder-oQ4e-mtp — used only as a structural comparison reference for expected tensor counts. No weights were derived from it.

Not affiliated with or endorsed by any of the above. All trademarks belong to their respective owners.

License

Apache-2.0, inherited from the source model. See the included LICENSE. Upstream notices and attributions apply to this derivative.