satgeze/Hy3-1M-GGUF

Hy3 (Hunyuan) 1M Context GGUF: first GGUF conversion of Tencent's 299B hy_v3, extended to a million tokens. Needle-certified, honestly labeled.

Status, precisely (updated July 9, 2026)

Needle-certified 10/10 at every rung from 262K to 786K (q8 KV, IQ2_M; Q4_K_M also 10/10 at 524K). At the full 1,048,576 window: 70 percent per-needle across three published seeds (9/10, 6/10, 6/10), so 1M is functional with real edge variance, not perfect. MTP speculative decoding was challenged during upstream review and survived a three-part correctness audit including a behavioral match against Tencent's official vLLM implementation (details below). These files still need the PR-branch build until llama.cpp PR #25395 merges. Every claim on this card links to raw evidence in this repo.

Tencent's Hy3 (299B MoE, ~17B active, hy_v3 architecture) quantized to GGUF and extended to a 1,048,576-token context with YaRN. As far as we can tell these are the first GGUF quants of Hy3 anywhere: at publish time (roughly 30 hours after Hy3's release) only MLX and MXFP4 quants existed, and mainline llama.cpp did not yet support hy_v3.

Honest status

• Architecture support: native support is proposed upstream in PR #25395 (under active review, including from an NVIDIA engineer). Until it merges, build the hy3-mtp branch below. The original community fork (charlie12345/ROCmFPX) remains a working alternative and is credited in the PR.
• Quants: Q2_K is a static quant (no importance matrix). IQ2_M and IQ1_M are importance-matrix quants - the imatrix was computed on our own Q2_K running RAM-resident, over a code-and-reasoning calibration set, then used to requantize from the archived f16. This is a bootstrap: a better imatrix could later be computed from a higher-fidelity runner.
• Long context, measured: full NIAH ladder published below (July 9, 2026): 10/10 at 262K, 393K, 524K and 786K; 70 percent per-needle at 1M over three seeds. Multi-hop chain retrieval 4/5 at 524K on both IQ2_M and Q4_K_M.
• MTP: working, audited. We ported hy_v3 MTP inference to llama.cpp ourselves. When upstream review questioned the acceptance behavior we ran a full correctness program: wiring verified line-by-line against the vLLM reference, quantization ruled out (f16 head behaves the same), and a behavioral cross-check against Tencent's official vLLM MTP on identical greedy prompts: vLLM 46.7 percent ungated acceptance vs our 46.9 percent, a dead match. Conclusion: the single-depth head is inherently confidence-gated; run it with --spec-draft-p-min 0.75 (88 percent acceptance, up to +37 percent decode on CUDA). See the MTP section below.
• Chat template: Hy3's stock template uses .format() calls that llama.cpp's Jinja engine rejects. chat_template_llamacpp.jinja in this repo is the fixed version; pass it with --chat-template-file.

What the gate actually produced (real, unedited)

Each quant was asked: "Name three colors and write a one-line haiku about the sea." Verbatim outputs, nothing cleaned up:

IQ1_M (62 GB, the roughest quant):

blue waves that whisper low, green depths that hold the silence of the sea.

IQ2_M (90 GB):

Three colors: blue, teal, and silver. waves drink the moonlight saltwind hums through silent blue- the sea forgets names

Q2_K (101 GB):

Blue, green, silver. The sea breathes in mist, silver light on rolling waves- quiet, endless blue.

My honest read

This is better than "it didn't collapse into repetition." A 1.7-bit quant of a 299B model (IQ1_M) still produces grammatical, on-theme lines; the two larger quants produce properly structured 5-7-5 haiku and correctly separate the color list from the poem. That is a genuinely good sign for how much of the model survives aggressive quantization here.

Two honest caveats, so you judge for yourself:

• This is a single short prompt. It proves coherence and basic instruction-following only. For reasoning and long-context retrieval, see the certification ladder above.
• IQ1_M is visibly the weakest: it dropped the explicit color list and gave two lines instead of a haiku. Usable, but the imatrix IQ2_M is the one I would actually run if the memory fits.

If your experience differs, the raw files are right here - try it and form your own view.

Needle-in-a-haystack certification (July 9, 2026)

Run on rented H200s (a 299B with full attention needs 86 GB of KV at 524K even at q8; no consumer card can host these rungs). Note the shape of the 1M column: the misses cluster at 25 to 45 percent depth, the classic lost-in-the-middle zone, while the edges stay solid. 10 needles per rung, depths 5 to 95 percent, temperature 0, seeded haystacks, q8 KV, MTP on with p-min 0.75. Raw records: results_niah_ladder.jsonl in this repo.

Rung	IQ2_M	Q4_K_M
262,144	10/10
393,216	10/10
524,288	10/10	10/10
786,432	10/10
1,048,576	9/10, 6/10, 6/10 (three seeds)

Reading the 1M row honestly: 21 of 30 needles over three independent seeds, misses at scattered depths, identical scores independently reproduced at q4 KV (9/10, same seed as the 9/10 run). Through 786K this model is needle-perfect; at the very edge of the baked window it retrieves about 70 percent per-needle. We publish that as "functional at 1M", not "certified at 1M", and both labels are earned.

Multi-hop tier (chained lookups, each answer keys the next, thinking ON): chains 4/5 at 524K on IQ2_M and 4/5 on Q4_K_M, 2/5 at 262K (seed variance). A harder multi-step variable-tracking tier is held back until our scorer for it is validated; unverified numbers do not get published here.

MTP: challenged in review, audited, confirmed correct

During upstream review an NVIDIA engineer questioned whether low ungated draft acceptance meant the MTP wiring was wrong. We ran the full program instead of arguing:

Test	Method	Result
Code parity	Our graph vs vLLM reference implementation, all four flagged axes (concat order, norm offset, pre/post-norm hidden, RMSNorm form)	Faithful on all four
Quantization ruled out	f16 full-precision head, greedy, ungated	46.9 percent (IQ2_M: 41.0), so quant is not the cause
Behavioral cross-check	Tencent's official vLLM MTP path, bf16, greedy, identical prompts, vLLM's own metrics	46.7 percent ungated: a dead match with our 46.9

Conclusion: Hy3's single-depth MTP head genuinely proposes low-confidence drafts when ungated, in every implementation including the official one. It is designed to be confidence-gated. Practical guidance, measured:

Condition (greedy)	Acceptance	Speed (2xH200, IQ2_M)
No speculation	n/a	72.2 tok/s
MTP ungated (p-min 0)	41.0 percent	73.6 tok/s (break-even)
MTP gated (p-min 0.75)	88.2 percent	81.5 tok/s (+13 percent)

On Apple silicon the gate is what keeps MTP from being a net loss; on CUDA ungated is break-even and the gate turns it into a gain. Earlier wording on this card described ungated MTP as a universal net slowdown; that was Metal-specific and is corrected here. Raw logs: results_mtp_audit/ in this repo.

Files

File	Size	Where	Notes
hy3-1M-MTP-IQ2_M.gguf	100 GB	HF + ModelScope	Recommended. The certified quant (table above), MTP baked
hy3-1M-MTP-Q2_K.gguf	111 GB	HF + ModelScope	Static quant + MTP
hy3-1M-MTP-IQ3_XXS.gguf	117 GB	HF + ModelScope	imatrix quant + MTP
hy3-1M-MTP-Q3_K_M.gguf	145 GB	HF + ModelScope	Sweet spot for 2x96GB rigs, MTP baked
hy3-1M-MTP-Q4_K_M.gguf	183 GB	HF + ModelScope (split parts)	Quality pick for 192GB+ rigs, 10/10 at 524K
hy3-1M-MTP-Q5_K_M.gguf	199 GB	HF + ModelScope (split parts)	Near-lossless tier
hy3-1M-MTP-Q6_K.gguf	~245 GB	HF + ModelScope (split parts)	Highest quant of the ladder, near-lossless
hy3-1M-IQ1_M.gguf	62 GB	ModelScope only	Smallest; fits a 128 GB Mac. no_think only; NOT needle-certified
hy3-1M-IQ2_M.gguf / hy3-1M-Q2_K.gguf	90 / 101 GB	HF + ModelScope	MTP-less originals, kept for history
chat_template_llamacpp.jinja	small	HF + ModelScope	Fixed template for llama.cpp

On ModelScope, files over the size cap arrive as -00001-of-0000N split parts; point llama.cpp at the first part and it finds the rest.

Field testing notes (real sessions, July 7)

Measured on a MacBook Pro M3 Max 128 GB, Metal, fork build, IQ1_M fully GPU-resident:

• Speed: 24-25 tok/s generation, ~41 tok/s prefill (llama-server timings, short prompts). A 299B MoE at chat speed on a laptop.
• no_think mode: coherent and usable, but noticeably chatty: filler sentences, over-explaining, occasional odd asides. Fine for casual chat and drafting; does not feel like a frontier chat tune.
• Thinking mode (reasoning_effort: high): collapses at IQ1_M. On a bare "hello" it produced hundreds of words of circular reasoning ("The user wrote hello... we should respond... but wait, the user wrote hello...") without ever converging to an answer. This is the 1.7-bit tax: short answers survive, reasoning chains do not. Do not use IQ1_M for coding or anything that needs thinking.
• Because of that, IQ1_M lives on ModelScope only; HF carries the quants we can recommend. IQ2_M field results will be added after the same test battery.
• Known cosmetic issue on the ROCmFPX fork build: the <|hy_eos:opensource|> stop token can leak as visible text at the end of responses. We root-caused it: the GGUF tokenizer metadata is correct (the token is properly CONTROL-typed); the leak is in that fork's chat parser, which renders special tokens for structural parsing and passes unknown trailing ones through. The recommended hy3-mtp branch build below does not have this issue (mainline strips specials before parsing). Workarounds on the fork: add the token as an explicit stop string, or apply the parser patches we are submitting to that repo. Generation always stops correctly; the text is just noise.

MTP speculative decoding: measured guidance (July 8)

The -MTP- quants carry Hy3's draft head (protected at q8_0). Measured on these exact files, upstream llama.cpp build (PR #25395):

Platform	MTP effect	Setting
CUDA (RTX 5090)	+26 to 37% decode	--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75
Apple silicon (M3 Max, IQ2_M)	neutral: 23.27 vs 23.21 tok/s, 88.7% acceptance	flags optional, no penalty with p-min 0.75

--spec-draft-p-min 0.75 is required either way: the MTP head is trained single-depth, and without the confidence gate speculation becomes a net slowdown (38.6% acceptance measured). Full data and discussion: Apple-silicon datapoint.

Speed expectations, honestly: quoted tok/s are fresh-context ceilings; in an ongoing conversation expect roughly 17 to 19 tok/s on the M3 Max at IQ2_M as context deepens. Both numbers are normal physics.

How to control thinking with llama.cpp: pass --chat-template-kwargs '{"reasoning_effort": "high"}' (or no_think / low) at server launch.

How to actually use a 1M-context model

Habits that measurably help, from our RULER, hop and adherence testing across this fleet:

1. Re-state standing instructions near the end of long prompts; recency beats depth.
2. One big reference dump beats a long accumulated conversation. Fresh session per task.
3. After any compaction or summarization, repeat your active rules yourself.
4. Prefill at 500K+ takes real time on any hardware; stage your questions accordingly.
5. Know your quant: IQ2_M holds retrieval to 786K; IQ1_M is a no_think chat quant, not a work quant.

Building an hy_v3-capable llama.cpp

First, check if you even need this: native support has been proposed upstream in ggml-org/llama.cpp#25395 (hy_v3 + MTP, AI-implemented with full disclosure, validated on these exact GGUFs). If that PR has merged by the time you read this, stock llama.cpp runs these files and you can skip everything below.

Until then, the recommended build is the branch that PR is made from: current llama.cpp master plus hy_v3 plus the MTP head. No web-UI workarounds needed, modern built-in UI included.

git clone --branch hy3-mtp --depth 5 https://github.com/satindergrewal/llama.cpp.git
cd llama.cpp

# Apple Silicon (Metal):
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

# NVIDIA (CUDA). If you build inside a conda env and hit
# "undefined reference to cuMemGetInfo", point at the real driver lib:
cmake -B build -DGGML_CUDA=ON \
  -DCUDA_cuda_driver_LIBRARY=/usr/lib/x86_64-linux-gnu/libcuda.so.1 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

MTP speculative decoding note for this model: launch with --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75. The p-min 0.75 matters: Hy3's MTP head is trained single-depth, and without the confidence gate speculation becomes a net slowdown (measured 38.6 percent acceptance vs 97.3 percent with it).

The original community implementation these quants were first built on is charlie12345/ROCmFPX (requires stubbing four empty web-UI files before building: tools/server/public/{bundle.css,bundle.js,loading.html,index.html}). It remains a fine option and is credited in the PR.

Run it (with an hy_v3-capable llama.cpp build)

llama-server -m hy3-1M-IQ2_M.gguf -c 262144 -np 1 --jinja \
  --chat-template-file chat_template_llamacpp.jinja

Start at 262K and raise -c toward 1048576 per available memory. On Apple Silicon the IQ1_M runs; on 192 GB+ the IQ2_M is the quality pick.

Credits

Base model: Tencent (Apache-2.0). hy_v3 llama.cpp implementation: community fork (charlie12345). GGUF conversion, 1M YaRN extension, imatrix bootstrap, and gating: SatGeze. Method and tooling: github.com/satindergrewal/aviary-1m.