prometheusAIR/Hy3-heretic-MTP-GGUF

Hy3-Heretic-295B-A21B — IQ2_XXS (attention-protected) with MTP restored

An imatrix GGUF quantization of trohrbaugh/Hy3-heretic, the decensored/abliterated variant of tencent/Hy3 (295B total / 21B active MoE).

Two things make this build unique:

1. It fits on a single 96 GB card with 64K context, fully resident in VRAM. The next-smallest llama.cpp-compatible Hy3-heretic quant is 100.01 GiB — larger than the card before you add any KV cache.
2. It has a working MTP head, so speculative decoding actually runs. No other Hy3-heretic release has this at any size, because they all inherit a source checkpoint whose MTP layer was lost. This one was grafted back.

---

Quick start

llama-server \
  --model Hy3-Heretic-MTP-IQ2_XXS-attnQ4K.gguf \
  -ngl 999 --ctx-size 65536 \
  -fa on -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 1024 \
  --jinja --chat-template-file hy3-chat-template.jinja \
  --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 \
  --temp 0.9 --top-p 1.0

Sampling defaults are per the Tencent model card (temp 0.9, top-p 1.0). Drop the --spec-* flags to run without speculative decoding.

---

Requirements

Needs a llama.cpp build from 2026-07-13 or later. hy_v3 support (with MTP) was added in PR #25395, merged 2026-07-13. Earlier builds will fail with unknown model architecture: 'hy_v3'. There is no ik_llama.cpp support for this architecture.

Use the fixed chat template, not the embedded one. The template inherited from the Tencent original renders the EOS token as literal text after assistant turns, which leaks <|hy_eos:opensource|> into content and breaks streaming tool calls. Pass a corrected jinja file with --chat-template-file.

---

Files

file	size
Hy3-Heretic-MTP-IQ2_XXS-attnQ4K.gguf	75.78 GiB (81,364,399,584 bytes)

Single file — 2.18 BPW, 1298 tensors, block_count 81, nextn_predict_layers 1.

hf download prometheusAIR/Hy3-heretic-MTP-GGUF \
  Hy3-Heretic-MTP-IQ2_XXS-attnQ4K.gguf --local-dir .

---

Fit on a single RTX PRO 6000 Blackwell (96 GB)

Measured usable VRAM on that card is ~95.6 GiB.

	weights	+ 64K q8_0 KV	+ compute	total	fits?
mradermacher Q2_K	100.01 GiB	—	—	over before KV	✗
this build	75.78 GiB	10.6 GiB	~2.5 GiB	~88.9 GiB	✓ (~6.7 GiB spare)

Measured resident footprint in practice: 89,854 MiB of 97,887.

---

Quantization recipe

Built with an imatrix (bartowski/Hy3-GGUF/Hy3-imatrix.gguf, 876 entries / 812 chunks). Bits are deliberately spent on the dense path rather than the experts:

tensor	type	why
attn_q, attn_k, attn_v, attn_output	Q4_K	attention is dense — every parameter is read for every token in all 80 layers, so its error compounds with depth
token_embd	Q8_0	it is a gather, not a matmul: higher precision costs ~nothing in speed
output	Q6_K	vocab-sized matmul each token
ffn_*_exps (routed experts)	IQ2_XXS	sparse — only 8 of 192 fire per token, so error is diluted and routing partially self-corrects
all of blk.80 (MTP layer)	Q4_K	see the imatrix note below

Census: 488 F32 / 467 IQ2_XXS / 331 Q4_K / 10 Q2_K / 1 Q6_K / 1 Q8_0. No iq1_s, iq2_s or iq3_s tensors — see the sm_120 warning below.

Why the dense path is protected — measured, not assumed

An earlier build of this same model used the stock llama-quantize IQ2_XXS recipe, which leaves attn_q and attn_output at ~2.06 bpw. Boosting only the dense path to the table above (+2.02 GiB, 2.09 → 2.15 BPW) produced a measurable behavioural change on a multi-step reasoning task with a 900-token budget:

build	result	length
stock recipe (2-bit attention)	truncated, no answer emitted	2613 chars
this recipe (Q4_K attention)	completed, correct answer	2178 chars

Cost: 9.1% throughput. Worth it.

---

MTP restoration

What was done: layer 80 was extracted from tencent/Hy3 (593 tensors, 3.753B params) and grafted into the abliterated weight set, taking the index from 46,545 to 47,138 tensors — exactly Tencent's own count. Trunk layers 0–79 are untouched abliterated weights; this was verified by sha256 against a non-MTP build of the same source (token_embd, output, and blk.0/40/79 attention and expert tensors are byte-for-byte identical).

The head is unmatched — Tencent's original MTP head paired with an abliterated trunk. This costs almost nothing:

prompt	acceptance here (unmatched)	reference: base Hy3, matched head
short reasoning	91.8%	87.1%
code	94.6%	93.5%
4K depth	90.0%	89.1%

Speculative decoding's acceptance test preserves the target model's sampling distribution, so a mismatched head affects speed, not quality.

Running with MTP

--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75

p-min 0.75 is required: Hy3's single-depth head is confidence-gated by design, and ungated acceptance is far lower in every implementation.

---

Measured performance

RTX PRO 6000 Blackwell 96 GB, fully VRAM-resident, 64K context, -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024:

prompt	with MTP	without MTP	gain
short reasoning	107.39 tok/s	96.70	+11.1%
code	121.11 tok/s	96.89	+25.0%
4K depth	88.27 tok/s	86.77	+1.7%
prompt processing @4K	~2420 tok/s

Cold load ~16 s from NVMe.

Correctness: needle-in-a-haystack exact at 1K / 4K / 16K / 32K / 56K tokens and at depths 0.25–0.9 (7/7, largest verified prompt 57,925 tokens). Streaming tool calls return finish_reason=tool_calls with correct arguments. Strict-JSON instruction following clean. Generated code passes executable doctests.

---

Limitations — please read

• The abliteration's effectiveness is UNVERIFIED here. A 6-prompt battery of legitimate requests that aligned models tend to over-refuse scored 0/6 refusals on this build — but also 0/6 on the non-abliterated base model, so the probe cannot discriminate. Whether the Heretic edit survived 2.18bpw quantization was not established. Evaluate it yourself against your own use case.
• It is roughly 50–70% more verbose than non-abliterated Hy3. On a task the base model finished in 945 characters, this build used ~1550–2180. Budget max_tokens generously; a truncated answer here is often a correct answer that ran out of room. Part of that verbosity was quantization (fixed by the attention boost above) and part is the abliteration itself.
• Structured reasoning is measurably weaker than base Hy3. On a bin-packing battery graded for conservation, capacity and optimality, abliterated builds score 2/5 where non-abliterated builds score 3/5 — consistent across two different quantization profiles, so this is an abliteration cost rather than a quantization one.
• This is a ~2.18 bpw quant of a 295B model. If it fits in your VRAM budget, a larger quant will be better.
• Inherits any biases and failure modes of tencent/Hy3 and of the Heretic abliteration process.

---

Notes for anyone reproducing this

Three things cost real time and are not obvious:

1. No imatrix covers blk.80. An imatrix is captured from ordinary forward passes and the MTP head never fires during those, so every imatrix for this architecture has that hole. Since ggml_quantize_requires_imatrix() is true for IQ2_XXS / IQ2_XS / IQ1_S, quantizing blk.80 to IQ2_XXS aborts the entire run at the last tensor, after roughly an hour of work. Force the layer to a type that needs no imatrix:

   --tensor-type 'blk\.80\.=q4_K'
   

   It must come first — overrides are std::regex_search with first-match-wins. (bartowski hit the same wall and solved it with Q4_0 for all of blk.80; that is the otherwise-inexplicable "11 Q4_0" in his IQ2_XXS census.)

2. --dry-run does not validate imatrix coverage. It computes target types and reports a size without running the imatrix check, so a clean dry-run is not a green light. Grep its output for blk.80 target types.

3. Do not copy shards wholesale when grafting. Layer 80 lives in 5 of Tencent's 99 shards, but 3 of those also contain 183 tensors belonging to layers 0–79. Copying those files in would silently revert those tensors to non-abliterated Tencent weights, partially un-abliterating the model with no error anywhere. Re-serialize only model.layers.80.*.

⚠️ Blackwell / sm_120 users

On the hardware this was built and tested on (compute capability 12.0 — RTX PRO 6000 Blackwell and RTX 5070 Ti), llama.cpp's CUDA matmul kernels for iq1_s, iq2_s and iq3_s produce garbage: verified with test-backend-ops, errors of 0.52–1.40 against a 5e-4 tolerance, on both MUL_MAT and MUL_MAT_ID. The failure is silent — no load error, just wrong output.

This matters because a quant's name does not tell you which ggml types are inside it. For example bartowski/Hy3-GGUF IQ2_XS contains 18 iq2_s tensors, and IQ2_M is built on iq2_s plus iq3_s.

This build deliberately contains none of those three types. Verified on one host and one llama.cpp build, so treat it as a strong warning rather than a universal claim — but if you are on Blackwell and a low-bit i-quant is producing nonsense, this is likely why.

---

Credits

• tencent/Hy3 — base model, and the source of the grafted MTP layer
• trohrbaugh/Hy3-heretic — the abliterated weights
• Heretic by p-e-w — the abliteration method
• bartowski — the importance matrix, and the Q4_0-on-blk.80 trick
• llama.cpp — and PR #25395 for hy_v3 + MTP support