xero0000/Kleiner-Coder-35B-vram13-MTP

🔬 Kleiner — Qwopus-Coder VRAM13 + Corpus-Tuned MTP

Black Mesa series · the coder, all-VRAM, with a finetuned multi-token head.

~13.5 GB all-VRAM GGUF of Jackrong's Qwopus-3.6-35B-A3B-Coder (agentic coding fine-tune of Qwen3.6-35B-A3B), with a corpus-finetuned MTP (nextn) head swapped in for self-speculative decoding under ik_llama.cpp (-mtp).

This is a separate release from the earlier native-head VRAM13 Qwopus build. Same backbone quant recipe; new blk.40 head trained on a held-out corpus (val loss ≈ 2.10, top-1 ≈ 62.2%), then exported into the GGUF.

TL;DR: coding model that fits dual mid-range GPUs (13.5 GB weights), runs with verification-lossless MTP decode at **144 t/s code / ~120 t/s prose** on a 3060 Ti + 3080 desktop. Thinking-off by design — keep --reasoning off for agent loops.

---

File

File	Size	Notes
Qwopus3.6-35B-A3B-Coder-vram13-q2ex-imat-MTP.gguf	~13.48 GB	VRAM13 backbone + corpus-tuned MTP head

Measured performance

Rig: RTX 3060 Ti 8 GB + RTX 3080 10 GB, DDR4, ik_llama.cpp, q4_0 KV, flash-attn, all layers on GPU, -mtp.

workload	decode	prefill
code generation (700 tok)	~144 t/s	~494 t/s
prose (500 tok)	~120 t/s	~526 t/s
@14.5K context (32 tok out)	~107 t/s	~1418 t/s

Speculative decoding verifies every draft token against the backbone, so output tokens match the non-MTP path; only throughput changes with acceptance rate (code usually accepts better than free prose).

Recipe

Backbone (VRAM13 / q2ex + imatrix)

Requantized from Q8 with a Qwopus-specific importance matrix (ik/TQ llama-quantize custom rules; nextn-aware — mainline can choke on the MTP layer):

• Experts blk 3–36 → Q2_K; edge experts 0–2 & 37–39 → Q3_K
• Attention + SSM projections → Q5_K
• Shared expert, output, embeddings → Q6_K
• Base ftype Q4_K so every custom rule fires

Principle: on small-active MoE, experts dominate size and tolerate 2-bit with imatrix; the every-token path stays higher precision.

MTP head (corpus-tuned)

The stock blk.40 nextn tensors were replaced (not grafted from another model) using a head trained offline against frozen backbone hidden states:

• Large MoE expert mats → Q4_0
• Attention / shared-expert / eh_proj → Q8_0
• Norms / router → F32, with Qwen3Next RMSNorm written as (1 + HF weight) (GGUF convention)
• ~0.50 GB head footprint

qwen35moe.block_count = 41, nextn_predict_layers = 1.

Training (MTP head only)

Only the nextn / MTP module is trained (~835M params). Backbone, embeddings, and lm_head stay frozen. The objective matches self-speculative decode: predict token t+2 from backbone hidden h_t and embed(t+1).

item	value
Corpus	local coding/agent session logs (Claude-style tool + chat traces), not a public web dump
Tokenized size	~1.66M tokens (ids.npy; 3,381 chunks of ≤512 tok)
Source text	~6.0 MB / ~72k lines of session transcript
Hidden dump	per-token h_t from the deployed quantized backbone (same distribution as runtime)
Windows	12,728 train + 664 val sequences × 128 positions (~5% val by chunk)
Schedule	2 epochs, cosine LR (peak 5e-5), micro-batch 2 × accum 16 (eff. 32), DDP on 2× consumer GPUs
Steps	794 optimizer steps
Baseline (stock head)	val loss 5.04, top-1 24.7%
Final (this head)	val loss 2.10, top-1 62.2%

Rough scale: ~1.7M tokens is a small, domain-specific finetune — enough to pull the draft head toward real agent/coding traces on this backbone, not a general pretrain. Speedups still come from draft acceptance; verified tokens are always the backbone’s.

How to run

Requires ik_llama.cpp for reliable nextn load + -mtp. Mainline may load the file but will not drive the head the same way.

Recommended (long agent / tool sessions — 64K)

Flash-attn + MTP KV can OOM mid-session at deeper contexts on ~18 GB dual-GPU rigs. 64K is the practical ceiling for sustained agent work on this hardware class:

./llama-server \
  -m Qwopus3.6-35B-A3B-Coder-vram13-q2ex-imat-MTP.gguf \
  --jinja --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on \
  --ctx-size 65536 --parallel 1 --n-gpu-layers 99 --ctx-checkpoints 8 \
  --tensor-split 44,56 --ubatch-size 512 \
  -mtp --ctx-size-draft 8192 \
  --reasoning off --reasoning-budget 0 \
  --no-mmap --threads 8 --no-warmup --port 8000

Short interactive / more headroom

You can try 128K with a smaller draft context and careful batch sizes; if you hit FA/KV OOMs under load, drop back to 64K or shrink --ctx-size-draft / --ubatch-size.

• Qwopus is a thinking-off design — keep --reasoning off (also avoids empty-content stalls in some clients).
• Single ≥16 GB GPU: drop --tensor-split.
• Drop -mtp if you only want the backbone (no speculative speedup).

Intended use & limitations

• Target: local agentic / tool-use coding with high decode speed on ~16–18 GB total VRAM.
• 2-bit experts are the quality floor vs larger mixed-q2k or Q8 builds; this file optimizes all-VRAM speed, not maximum fidelity.
• MTP acceptance (and thus speedup) varies by domain; code-like text is usually strongest.
• Inherits capabilities and biases of the Qwopus-Coder fine-tune. Head swap does not change verified next-token distribution of the backbone.

Related

• Parent coding fine-tune: Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF
• Mixed q2_K + imatrix (CPU offload, 256K): Kleiner-Coder-35B-mixed-q2k
• Base Qwen VRAM13 (non-coder): see other xero0000 VRAM13 uploads

Provenance

• Base: Qwen3.6-35B-A3B (Alibaba / Qwen, Apache-2.0)
• Coding fine-tune: Jackrong (Qwopus-3.6-35B-A3B-Coder)
• VRAM13 quant + imatrix, corpus MTP train/export, benches: xero0000, July 2026

Released under Apache-2.0 (same family as the base).