Qwimi-3.6-27B-Coder-MTP-GGUF

GGUF quantizations of Qwimi-3.6-27B-Coder-MTP-BF16 — a coding-focused SFT of Qwen 3.6 27B (mixed coding + tool-calling + agentic training; see Qwimi-3.6-27B-BF16 for the full model card, training configuration, dataset composition, and eval results). This repo exists for llama.cpp and downstream tools (Ollama, LM Studio, text-generation-webui, etc.).

Status: public repository. All 10 planned quants are uploaded and confirmed.

---

Contents

1. Source & conversion
2. Available quants
3. Which quant should I use?
4. Important: llama.cpp version requirement
5. How to run
6. Tool-calling with llama.cpp
7. Training summary
8. Known limitations
9. Benchmarks

---

1. Source & conversion

• Source: trjxter/Qwimi-3.6-27B-BF16 (merged full-precision model)
• Converted with: llama.cpp — convert_hf_to_gguf.py --outtype bf16, then llama-quantize per target quant
• Build: CPU-only (GGML_CUDA=OFF); quantization is a CPU-bound process and does not require GPU
• Conversion note: the merged repo's tokenizer_config.json originally carried a transformers-v5-internal tokenizer_class value that AutoTokenizer (used internally by the converter) could not resolve. This was corrected before conversion; if you reproduce this conversion yourself from an older snapshot of the BF16 repo, verify tokenizer_config.json reflects the base model's tokenizer class first.

2. Available quants

All sizes confirmed from the uploaded files (Hugging Face Hub hardware-compatibility widget, computed from actual repo blobs). Total repo size: 175.0 GB across 10 quants.

File	Type	Size	Status
Qwimi-3.6-27B-Q2_K_L.gguf	Q2_K (Q8_0 embed/output tensors)	12.1 GB	✅ confirmed
Qwimi-3.6-27B-Q3_K_S.gguf	Q3_K_S	12.3 GB	✅ confirmed
Qwimi-3.6-27B-Q3_K_M.gguf	Q3_K_M	13.5 GB	✅ confirmed
Qwimi-3.6-27B-Q3_K_L.gguf	Q3_K_L	14.6 GB	✅ confirmed
Qwimi-3.6-27B-Q4_K_S.gguf	Q4_K_S	15.8 GB	✅ confirmed
Qwimi-3.6-27B-Q4_K_M.gguf	Q4_K_M	16.8 GB	✅ confirmed
Qwimi-3.6-27B-Q5_K_S.gguf	Q5_K_S	19.0 GB	✅ confirmed
Qwimi-3.6-27B-Q5_K_M.gguf	Q5_K_M	19.5 GB	✅ confirmed
Qwimi-3.6-27B-Q6_K.gguf	Q6_K	22.4 GB	✅ confirmed
Qwimi-3.6-27B-Q8_0.gguf	Q8_0	29.0 GB	✅ confirmed

Q2_K_L naming note: this is not a standard llama-quantize type — it's the community recipe of a Q2_K base with the token-embedding and output (LM head) tensors held at Q8_0 precision (--token-embedding-type Q8_0 --output-tensor-type Q8_0), which meaningfully improves quality over plain Q2_K at a small size cost. All other quants above are standard llama.cpp K-quant types.

Not produced: Unsloth "UD" (Dynamic 2.0) quants and sub-3-bit i-quants (IQ2_*, IQ3_XXS) are not included — these require Unsloth's proprietary calibrated quantization pipeline, not plain llama-quantize, and are out of scope for this repo.

3. Which quant should I use?

General guidance for this model family and size (27.8B):

• Best balance of quality/size: Q4_K_M — the standard recommended default for most use.
• Near-lossless, if you have the VRAM/RAM: Q8_0 (30 GB) or Q6_K (23 GB).
• Constrained VRAM: Q4_K_S or Q3_K_M/Q3_K_L — expect a noticeable but usually acceptable quality drop versus Q4_K_M.
• Extreme constraint: Q2_K_L — usable, but at 2-bit the model's coding accuracy and instruction-following will degrade more than the size savings suggest; only recommended if no larger quant fits your hardware.

This model was trained and merged in BF16; no quant-specific fine-tuning was performed, so relative quality ordering follows standard llama.cpp quantization behavior for models this size — it has not been separately benchmarked per quant here (see §9).

4. Important: llama.cpp version requirement

Qwen3.6-27B (model_type: qwen3_5) is a hybrid architecture — standard attention layers interleaved with GatedDeltaNet linear-attention layers. Support for this architecture in llama.cpp and downstream tools (Ollama, LM Studio, text-generation-webui) depends on how recently they've pulled in qwen3_5/GatedDeltaNet support upstream. Verify your llama.cpp build (or wrapping tool) explicitly supports this architecture before assuming compatibility — an outdated build may fail to load the model or silently misinterpret the linear-attention layers. If you hit load errors, check for a llama.cpp update first.

5. How to run

# llama.cpp CLI
./llama-cli -m Qwimi-3.6-27B-Q4_K_M.gguf -p "Write a Python function to reverse a linked list." -n 512

# llama-server (OpenAI-compatible API)
./llama-server -m Qwimi-3.6-27B-Q4_K_M.gguf --ctx-size 16384

Set --ctx-size to at most 16384 — that is the sequence length this model was trained and validated at, even though the base architecture supports a much longer native context. Behavior beyond 16,384 tokens is unvalidated.

6. Tool-calling with llama.cpp

This model uses Qwen 3.6's native XML-style tool-call format (<tool_call><function=...> <parameter=...>), not the JSON-object format from earlier Qwen versions — see the full example in the Qwimi-3.6-27B-BF16 README §6. For llama-server to apply the model's chat template (including tool definitions) correctly, you need a recent build with Jinja chat template support enabled (commonly a --jinja flag) — verify against your installed llama.cpp version's documentation, since flag names and defaults change across releases.

7. Training summary

Condensed from the full model card (Qwimi-3.6-27B-BF16) — see that repo for complete details, dataset composition, and eval curves.

• Base: unsloth/Qwen3.6-27B, dense 27.8B, text-only SFT (vision frozen)
• Method: 4-bit QLoRA, r=64/alpha=64, language layers only
• Data: mixed single-run SFT — coding (82.5% of tokens) + tool-calling (4.8%) + agentic SWE-agent trajectories (12.7%), 22,359 total training rows, 72.5M training tokens
• 1 epoch, effective batch 16, 1,398 optimizer steps, A100 80GB
• Final eval loss: coding 0.4761, tool 0.0208, agentic 0.2892 — all three domains improved monotonically through the full run with no overfitting signal

8. Known limitations

See Qwimi-3.6-27B-BF16 §9 for the full list (vision untouched, thin agentic coverage, one capped low-quality coding source, 3.6% of coding examples dropped for length rather than truncated, 16,384-token validated context). Additionally for this GGUF repo specifically:

• Quantization introduces its own quality loss on top of the base model's characteristics, increasing with lower bit-width — see §3 for guidance.
• This is a fresh/bleeding-edge architecture in llama.cpp (§4) — expect rougher edges in tooling support than for long-established architectures.

9. Benchmarks

Coming soon.