fetchingcat/occamy-1.0-mtp-gguf

Occamy-1.0 MTP — GGUF (Q4_K_M, imatrix)

Accio-Lab/occamy-1.0 quantized to Q4_K_M with an importance matrix, with multi-token prediction (MTP) restored so llama.cpp and Ollama can run speculative decoding.

Vision is included via a separate mmproj file.

Why this repo exists

Occamy-1.0 is a full-parameter fine-tune of Qwen/Qwen3.6-35B-A3B. The base checkpoint ships an MTP head; Occamy does not. Its config declares text_config.mtp_num_hidden_layers = 0 and the release contains zero mtp.* tensors. Every published Occamy GGUF is therefore a straight conversion that cannot do speculative decoding:

Repo	Parameters in GGUF
bartowski/Accio-Lab_occamy-1.0-GGUF	34,660,610,688
mradermacher/occamy-1.0-i1-GGUF	34,660,610,688
this repo	35,505,251,456

The difference is exactly 844,640,768 parameters — the MTP head.

What was actually done

The 19 mtp.* tensors from Qwen/Qwen3.6-35B-A3B (revision 995ad96e) were copied into the Occamy checkpoint and mtp_num_hidden_layers was set back to 1. Occamy keeps the base architecture unchanged (40 layers, 256 experts, hidden 2048, vocab 248320), so the head drops straight in. convert_hf_to_gguf.py then maps it to blk.40.nextn.* with block_count=41 and nextn_predict_layers=1.

Please read this part. The MTP head was trained by Qwen to draft against the base model's hidden states. Occamy then changed those hidden states through SFT, HDPO, model merging and SAO. This head was not trained for Occamy — it is transplanted, and it drafts slightly worse here than it does on base Qwen3.6.

This is safe rather than merely acceptable: llama.cpp verifies every drafted token against the full model and discards rejects. A poorly-matched draft head costs throughput, not output quality. It cannot make the model say something the model would not otherwise say.

Measured

Acceptance is content-dependent (measured in Ollama, --spec-draft-n-max 2):

Content	mean acc len	acc rate/pos
Prose / general reasoning	2.22 – 2.29	(0.73, 0.49) – (0.75, 0.54)
Structured text, arithmetic	2.69 – 2.81	(0.91, 0.78) – (0.96, 0.86)

Throughput A/B on identical prompt and settings (llama-cli, DGX Spark GB10, Q4_K_M):

	tokens/s
--spec-type none	43.5
--spec-type draft-mtp	58.9

≈1.35× speedup.

For reference, the same head on its native base model reaches ~2.6 mean acceptance length, so roughly 85–90% of native acceptance survives the transplant.

Vision

Occamy's vision tower was verified bit-identical to Qwen/Qwen3.6-35B-A3B — all 333 tensors compared equal. Accio froze the encoder and projector during SFT, and that is literally true in the weights.

Note that the projector was aligned to the base language model, which Occamy's training moved. This is a property of the upstream release, not of this conversion, and applies equally when serving Occamy via vLLM or SGLang. Accio publishes no vision benchmarks.

Files

File	Size	Purpose
occamy-1.0-mtp-Q4_K_M.gguf	21.7 GB	The model
mmproj-occamy-1.0-F16.gguf	899 MB	Vision projector (optional)
model-mtp.safetensors	1.57 GB	The 19 MTP tensors, for reproducing the graft
occamy-mtp-transplant.py	—	Script that performs the graft
occamy-1.0-imatrix.dat	192 MB	Importance matrix (GGUF format, despite the suffix)

model-mtp.safetensors and the script are included so you can build the grafted checkpoint yourself at full precision — useful for vLLM/SGLang, or for making your own quants — and so the claims above are independently verifiable.

Usage

llama.cpp

llama-cli -m occamy-1.0-mtp-Q4_K_M.gguf \
  -ngl 99 -c 262144 -fa on -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 2 --jinja

Add --mmproj mmproj-occamy-1.0-F16.gguf for images.

Ollama

FROM ./occamy-1.0-mtp-Q4_K_M.gguf
FROM ./mmproj-occamy-1.0-F16.gguf
PARAMETER num_ctx 262144
PARAMETER num_predict 32768
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 20
PARAMETER presence_penalty 1.5
PARAMETER draft_num_predict 2

Sampling values follow Accio's recommendations for Occamy. presence_penalty 1.5 matters — without it the model can loop in long reasoning traces. num_predict bounds runaway <think> blocks; it caps generation length, not context.

262144 is the model's native context window and was tested here end to end with a q8_0 KV cache. Only 10 of the 40 layers carry a growing KV cache — every fourth layer (3, 7, 11 … 39); the other 30 are linear-attention layers with constant state. The cache is therefore roughly a quarter the size of a conventional 40-layer model at the same length, which is what makes the full window practical. Lower num_ctx only if you are memory constrained.

On temperature

Accio recommends 1.0 for Occamy, which is what is set above. Qwen's guidance for the Qwen3 thinking family is 0.6, and some Qwen3.6 builds ship that value instead. Either is defensible; 0.6 gives less variance.

If you do lower it, note that Accio tuned temperature and presence_penalty 1.5 as a pair. A penalty that aggressive was chosen to tame sampling at 1.0, so at 0.6 you may want to ease it back toward 0.5.

Confirm MTP is live:

docker logs ollama 2>&1 | grep "statistics.*draft-mtp"

Quantization

imatrix computed with llama-imatrix over calibration_datav3.txt (129 chunks, 512-token context). Per-expert coverage was 99.2–99.6% across the 256-expert layers.

The MTP head (blk.40.*), token_embd.weight and output.weight receive no imatrix data — 13 tensors fall back to default Q4_K_M rules. Layer 40 is inactive during a normal forward pass, so no calibration statistics exist for it. This is expected.

Built with llama.cpp b10687 (c841aeeb8).

Reproducing

hf download Accio-Lab/occamy-1.0
hf download Qwen/Qwen3.6-35B-A3B --revision 995ad96eacd98c81ed38be0c5b274b04031597b0

python occamy-mtp-transplant.py --out ./occamy-1.0-mtp

python convert_hf_to_gguf.py ./occamy-1.0-mtp --outfile occamy-BF16.gguf --outtype bf16
llama-quantize --imatrix occamy-1.0-imatrix.dat occamy-BF16.gguf occamy-Q4_K_M.gguf Q4_K_M

Where each piece comes from:

• occamy-mtp-transplant.py — in this repo. It reads both checkpoints from your HuggingFace cache (HF_HOME / HF_HUB_CACHE are honoured) and writes the grafted directory.
• convert_hf_to_gguf.py, llama-quantize, llama-imatrix — from llama.cpp, build b10687 (c841aeeb8) or newer. Qwen3.6 MTP support is recent, so older builds will drop the head.
• occamy-1.0-imatrix.dat — in this repo, so the calibration pass is optional.

To regenerate the importance matrix instead of using the included one:

llama-imatrix -m occamy-BF16.gguf -f calibration_datav3.txt \
  -o occamy-1.0-imatrix.dat -ngl 99 -c 512 -b 512 -ub 512 -fa on

calibration_datav3.txt is not redistributed here; it is bartowski's public calibration set.

The transplant script symlinks Occamy's unchanged shards, so the grafted directory costs about 1.6 GB rather than a second full copy of the model.

What this is not

• Not benchmarked for quality. No IFEval, BFCL, Claw-Eval or similar were run. Accio's published scores are for their bf16 release, not for this 4-bit quantization.
• Not an official release. Not affiliated with or endorsed by Accio-Lab or the Qwen team.
• Not a retrained MTP head. It is Qwen's base head, transplanted. A head actually trained on Occamy's hidden states would very likely accept better.

Credits and licensing

• Qwen/Qwen3.6-35B-A3B — Qwen team, Apache-2.0. Source of the MTP head and the vision tower.
• Accio-Lab/occamy-1.0 — Accio Lab, Apache-2.0. The fine-tuned model.

Both upstream works are Apache-2.0; this repository is released under the same license.