isneezekittens/Carwin-28B-MTP-GGUF

Carwin-27B-GGUF-MTP

A dense 27B local model: a DARE-TIES merge of reasoning-heavy Darwin and agent/tool-calling-heavy Carnice, on the Qwen3.6-27B base, converted to GGUF with the Qwen3.6 MTP (multi-token prediction) head preserved in-file for self-speculative decoding in llama.cpp.

Why this exists

Built by a tech/AI hobbyist who enjoys tinkering with the Hermes agent framework. The goal was personal and specific: combine the tool-calling strength of Carnice with the reasoning of Darwin into one local, private model. This GGUF is the llama.cpp-format version, made to share and to keep a portable, runtime-agnostic copy. (An MLX build of the same model exists for Apple Silicon / oMLX.)

What it is

Base	Qwen/Qwen3.6-27B
Reasoning parent	FINAL-Bench/Darwin-28B-Opus
Agent / tool-calling parent	kai-os/Carnice-V2-27b
Merge method	DARE-TIES (50/50, density 0.53 each, BF16)
Format	Q4_K_M GGUF, MTP head preserved (Q8_0 / F32)
Companion	optional vision projector (mmproj), untested on this merge
License	Apache-2.0 (all three parent lines permissive)

How it was built

Built entirely on a single 32GB Mac Studio (M2 Max), agent-driven through the Hermes framework using a mix of MiMo v2.5 and GPT-5.5 — no cloud GPUs or rented compute.

The model was produced from a full-precision BF16 master: a DARE-TIES merge of Darwin and Carnice against the Qwen3.6-27B base, with the 15-tensor Qwen3.6 MTP head grafted in. GGUF and MLX are separate branches off that one master — GGUF is not converted from MLX.

GGUF path:

1. Build mainline llama.cpp (Metal, build tag b9601 — past the b9418 minimum for reliable Qwen3.6 MTP). Verified in the actual binary that --spec-type draft-mtp is supported.
2. Convert the BF16 master to an F16 GGUF with convert_hf_to_gguf.py (default bundles the MTP head; --no-mtp not passed). Result: 866 tensors, 54.6 GB.
3. Verify the MTP head survived conversion by inspecting the GGUF tensor list (the mtp.* head maps to blk.64.nextn.*, plus the qwen35.nextn_predict_layers metadata key).
4. Quantize the body to Q4_K_M while pinning the draft-head tensors high: --tensor-type "nextn=Q8_0". The body is Q4_K_M; blk.64.nextn.eh_proj stays Q8_0 and the nextn norms stay F32, so the draft head isn't degraded.
5. Verify again that the nextn tensors and their high precision survived quantization, by tensor-list inspection — not by assuming the command worked. Silent MTP drop is the known failure mode for this kind of build, so presence is always confirmed by reading the actual tensor list.

Files

File	Purpose
carwin-Q4_K_M.gguf	Main model. Q4_K_M body, MTP head at Q8_0 / F32.
carwin-mmproj-f16.gguf	Optional vision projector. Only needed for image input. Untested on this merge.

Running (llama.cpp)

Requires a recent llama.cpp build (b9418 or newer) for Qwen3.6 MTP support.

llama-server -m carwin-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 1 \
  -ngl 99 \
  -c 8192

Notes:

• Always set an explicit -c. Recent llama.cpp builds default to the model's full context window; the resulting KV cache can exhaust system memory.
• Try --spec-draft-n-max 1 first. Dense ~27B models can regress at higher draft values — benchmark 1 vs 3 on your own hardware.
• Because the body is a merge that drifted from stock Qwen3.6 while the MTP head comes from the base, draft acceptance may run lower than stock Qwen3.6. Coherent output with non-zero acceptance is the working bar.
• Vision: add --mmproj carwin-mmproj-f16.gguf for image input. Untested on this merge; vision and MTP may not combine in a single session.

Validation

• MTP head verified present and high-precision in both the F16 and the Q4 GGUF, by tensor-list inspection.
• Reasoning: the bat-and-ball problem is answered correctly with clean step-by-step working (the ball costs $0.05).

Performance (tokens/sec, draft-acceptance rate) has not been benchmarked under controlled conditions and is intentionally not stated here. Measure it on your own hardware.

Known quirks

• Identity: the model may identify as Gemini. This is cosmetic lineage residue from the merge, not a fault.
• Dense 27B: thorough and local, not small-and-fast.
• MTP preserved, acceptance unproven on this merge: the head is physically in the file; how well speculative decoding accepts on the merged body is for you to measure.
• Vision is untested on this merge.

Credits

All credit to the authors of the parent models and base: FINAL-Bench/Darwin-28B-Opus, kai-os/Carnice-V2-27b, and Qwen/Qwen3.6-27B. Merged with mergekit; MTP support via llama.cpp PR #22673.