knoopx/Ornith-1.5-35B-A3B-NInfer

Ornith-1.5-35B-A3B-MTP — NInfer Artifact Card

This card documents the derived NInfer .ninfer inference artifact for the ornith-ai/Ornith-1.5-35B-A3B model, not the upstream Hugging Face repository itself.

Ornith-1.5-35B-A3B is a Qwen 3.5 Mixture-of-Experts (MoE) multimodal language model (text + image input) with a DFlash multi-token-prediction (MTP) draft head. The upstream repo publishes BF16 safetensors (17 shards, ~143.84 GB). This artifact is a single groupwise-quantized NInfer bundle built from that BF16 source, intended for serving through the NInfer engine.

This MTP variant replaces the shipped Ornith-1.5 MTP head — which was untrained (random-init, 21–32% draft acceptance) — with a trained MTP head from shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY. In our measured comparison on a prose prompt (this host, RTX 5090, single-stream CLI, spec flags from models.nix), the trained head reaches 39.0% overall draft acceptance vs 21.8% for the native head — a **1.8×** improvement and +21.5% decode throughput (336.5 vs 277.0 tok/s). See the Benchmark section.

Scope note. This is a derived inference artifact. The upstream ornith-ai/Ornith-1.5-35B-A3B repository remains the source of truth for the model weights and the canonical (MIT-licensed) distribution. What is described here is a derived, quantized, engine-specific container produced by the conversion run.

Model Architecture

Property	Value
Architecture	qwen3_5_moe (Qwen 3.5 MoE)
Internal model id	qwen3.6-35b-a3b
Hidden layers	40
Experts	256 (8 active per token, A3B ≈ 3B active)
Hidden size	2048
Attention	GQA — 16 heads : 2 KV heads (head_dim 256)
Vocabulary size	248,320
Max position	262,144
Modality	Multimodal (text + image input); 27-layer vision encoder (hidden 1152, patch 16, out_hidden 2048)
MTP / draft	DFlash draft model — z-lab/Qwen3.6-35B-A3B-DFlash (DFlashDraftModel, 6 layers, 32/8 heads)
MTP head	Trained 15-tensor head from shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY (897,934,336 bytes, KL-distilled from the Qwen3.6-35B-A3B MTP head)
Base pin for front-end resources	Qwen/Qwen3.6-35B-A3B
License	MIT (base); Apache-2.0 (grafted MTP head, Qwen-derived)

The NInfer Artifact

The single self-contained inference bundle produced from the BF16 source:

Property	Value
Filename	Ornith-1.5-35B-A3B-MTP-w8g32-q4g64-q5g64-q6g64-bf16.ninfer
Size	22,783,245,824 bytes (21.22 GiB)
SHA-256	178fbdf85758af554578b0051372b8f1523bd4be677abac139c9d7322a5b8ac9
Objects	940 (tensors 934 + resources 6)
model_id	qwen3.6-35b-a3b
weights_id	groupwise-int
Recipe	qwen3_6_35b_a3b-v2

Quantization scheme

The artifact uses groupwise mixed-precision integer quantization, applied on-the-fly at conversion time (Q4/Q5/Q6/W8). It is a mixed-precision layout: the four integer group-wise formats carry the bulk of the expert/MLP weights, while a set of tensors is kept in full BF16 precision.

Weight types present in the artifact:

Weight type	Description	Notes
w8g32	8-bit weights, 32-element group scale	integer, group-wise
q4g64	4-bit weights, 64-element group	integer, group-wise
q5g64	5-bit weights, 64-element group	integer, group-wise
q6g64	6-bit weights, 64-element group	integer, group-wise
bf16	Full precision	embeddings, layer norms, attention output projections, the grafted MTP head

Format histogram

Format	Count
BF16	487
FP32	60
I32	1
W8G32_F16S	195
Q4G64_F16S	95
Q5G64_F16S	91
Q6G64_F16S	5

(934 tensors total; the four *_F16S group-wise counts together account for the 386 quantized tensors.)

Component tensor bytes

Component	Bytes
main_text	21,038,461,952
mtp	897,934,336
vision	280,664,992
dflash	410,053,632
draft_head	143,130,624
total	22,770,245,536

Provenance / Conversion

This artifact was produced by the NInfer source converter (the NInfer converter source tree), running in groupwise-int4 mode (i.e. on-the-fly group-wise quantization at conversion time) using the recipe qwen3_6_35b_a3b-v2.

Converter

• Module: tools.convert.qwen3_6_35b_a3b.convert (in the NInfer converter source tree)
• Recipe: qwen3_6_35b_a3b-v2 (recipe id RECIPE_ID = "qwen3_6_35b_a3b-v2")
• Quantization mode: groupwise-int4 (Q4/Q5/Q6/W8 on the fly)
• Converter revision: a9136e412b8418a8189152e91fdd571044d0450b
• Sources:
  • Base model: ornith-ai/Ornith-1.5-35B-A3B (BF16 safetensors, 17 shards)
  • DFlash draft model: z-lab/Qwen3.6-35B-A3B-DFlash
  • Trained MTP head (graft): shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY — 15 mtp.* tensors, 897,934,336 bytes. Initialized from the Qwen/Qwen3.6-35B-A3B MoE MTP head, then re-aligned to Ornith-1.5 hidden states via full-vocabulary KL distillation (two-epoch code-heavy warm-start).

Conversion command

From the NInfer converter source tree:

PYTHONPATH=$PWD python -m tools.convert.qwen3_6_35b_a3b.convert \
  --model /home/knoopx/models/llm/ornith/ornith-ai-Ornith-1.5-35B-A3B-base \
  --dflash-model /home/knoopx/models/llm/ornith/z-lab-Qwen3.6-35B-A3B-DFlash \
  --out out/ornith-1.5-35b-a3b-mtp.ninfer --device cpu

• Device: cpu
• Elapsed: 228.1 s

Front-end Resources

The bundle embeds the model's own 6 front-end resources (its own chat_template.jinja and the other 5):

1. tokenizer.json
2. tokenizer_config.json
3. generation_config.json
4. chat_template.jinja
5. preprocessor_config.json
6. video_preprocessor_config.json

Note. Unlike the base knoopx/Ornith-1.5-35B-A3B-NInfer artifact (which embedded the base pin's chat_template.jinja), this artifact embeds the model's own chat_template.jinja — prompt formatting behaviour matches the model's native template.

Size delta. This artifact (22,783,245,824 bytes) is 256 bytes smaller than the existing untrained-MTP artifact ornith-ai/Ornith-1.5-35B-A3B-w8g32-q4g64-q5g64-q6g64-bf16.ninfer (22,783,246,080 bytes). The delta is the size difference between the model's own chat_template.jinja (embedded here) and the base pin's chat_template.jinja (embedded in the untrained-MTP artifact).

Benchmark

Measured on this host (RTX 5090, sm_120a), single-stream CLI, identical spec flags for both artifacts (--spec mtp --draft-tokens 5 --lm-head-draft --draft-adaptive --draft-adaptive-min 3 --prefill-chunk 4096 --vision), a 150-word prose prompt, --max-new 256, reproducible sampling (--temperature 0.6 --top-k 20 --top-p 0.95 --seed 0). The unfixed artifact is the native ornith-ai head; the fixed artifact is this card's trained shisa-ai head.

Metric	Unfixed (ornith-ai)	Fixed (shisa-ai)	Δ
MTP acceptance rate	21.79%	39.03%	+17.2 pts (1.79×)
MTP accepted tokens	100	137	+37
MTP drafted tokens	459	351	−108
MTP rounds	154	117	−37
MTP acceptance length	1.65 tok/round	2.17 tok/round	+0.52
MTP fallback steps	1	1	0
Decode tok/s (spec)	276.95	336.51	+59.56 (+21.5%)

The fixed head drafts fewer tokens (351 vs 459) across fewer rounds (117 vs 154) but accepts more of them per round (2.17 vs 1.65 tok/round) — the signature of a stronger draft head that the target verifies more often, with less wasted speculative work.

Note. The shisa-ai head was trained with a code-heavy distribution, so the ~69% code / ~60% overall acceptance figures from the upstream repo are expected on code prompts. The numbers above are a prose prompt (the harder case for a code-tuned head) and therefore land lower; a code-generation prompt should approach the upstream figures. These prose numbers are the validated, reproducible baseline.

Serving

• Engine: NInfer — a custom C++ inference engine, built for CUDA 13, sm_120a.
• Served via: the NInfer engine.
• Model id: shisa-ai/Ornith-1.5-35B-A3B-MTP
• Capabilities: multimodal (text + image input); MTP (multi-token prediction) via the trained MTP head and the DFlash draft head.

Serving flags

--max-concurrency 2 \
--kv-capacity auto \
--kv-dtype int8 \
--spec mtp \
--draft-tokens 5 \
--lm-head-draft \
--draft-adaptive \
--draft-adaptive-min 3 \
--prefill-chunk 4096 \
--vision

Flag	Meaning
--max-concurrency 2	Two concurrent requests
--kv-capacity auto	Automatic KV-cache sizing
--kv-dtype int8	INT8 KV cache
--spec mtp	Speculative decoding via the MTP/draft head
--draft-tokens 5	Draft tokens per step
--lm-head-draft	Draft head uses the LM head
--draft-adaptive / --draft-adaptive-min 3	Adaptive draft budget (minimum 3)
--prefill-chunk 4096	4096-token prefill chunking
--vision	Enable the vision (image-input) path

Known Issues & Limitations

• No quality benchmark run. No KL-divergence or perplexity comparison has been run on this artifact. Quantization fidelity is not validated by a benchmark.
• Not numerically validated. No smoke/quality run against this artifact has been completed; treat quantization fidelity as unmeasured.
• Speculative-decoding output correctness on the trained head was not independently re-validated for this artifact.
• Template: N/A — the model's own chat_template.jinja is embedded (no mismatch).

License

The model base is released under the MIT license (the front-matter license: mit field follows the knoopx artifact-repo convention for the derived container). The grafted trained MTP head is a derivative of the Apache-2.0 Qwen/Qwen3.6-35B-A3B MTP head. The quantization/conversion tooling (the NInfer inference engine) governs any additional license terms on the engine itself.

Citation / Provenance

• Upstream repo: ornith-ai/Ornith-1.5-35B-A3B (BF16 source)
• Trained MTP head: shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY
• DFlash draft: z-lab/Qwen3.6-35B-A3B-DFlash
• Base front-end pin: Qwen/Qwen3.6-35B-A3B
• Inference engine: NInfer (CUDA 13, sm_120a)
• Converter: tools.convert.qwen3_6_35b_a3b.convert (recipe qwen3_6_35b_a3b-v2), groupwise-int4 mode